Oct 29 21:41:40.918201 ip-10-0-141-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:41:41.304225 ip-10-0-141-100 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:41.304225 ip-10-0-141-100 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:41:41.304225 ip-10-0-141-100 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:41.304225 ip-10-0-141-100 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307485 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307570 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307574 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307577 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307579 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.307574 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307583 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307586 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307589 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307592 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307595 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307598 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307601 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307603 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307606 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307610 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307615 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307618 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307621 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307624 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307628 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307632 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307635 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307644 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307647 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.307788 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307650 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307653 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307656 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307659 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307661 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307664 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307667 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307669 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307672 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307675 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307677 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307680 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307682 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307685 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307688 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307691 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307694 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307697 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307699 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307702 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.308249 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307705 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307708 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307710 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307713 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307716 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307718 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307721 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307723 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307726 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307729 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307732 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307735 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307737 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307740 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307742 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307745 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307747 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307750 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307752 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307755 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.308711 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307757 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307760 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307762 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307765 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307767 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307770 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307772 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307775 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307779 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307781 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307784 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307786 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307789 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307792 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307795 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307797 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307801 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307803 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307806 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.309192 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307808 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307811 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307813 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307816 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307818 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307822 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307825 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307828 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307830 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307832 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307835 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307837 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307840 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307843 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307845 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307848 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307852 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307855 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307858 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.309642 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307861 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307865 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307870 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.307873 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309525 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309534 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309537 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309540 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309543 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309545 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309548 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309551 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309555 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309557 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309561 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309564 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309568 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309571 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309576 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.310150 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309580 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309584 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309587 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309590 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309593 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309595 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309598 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309600 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309603 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309606 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309608 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309611 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309613 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309615 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309619 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309622 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309624 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309627 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309629 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.310629 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309632 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309635 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309638 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309640 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309643 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309645 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309648 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309651 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309653 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309656 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309658 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309661 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309664 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309666 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309669 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309671 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309674 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309677 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309679 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309681 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.311094 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309684 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309686 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309689 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309691 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309694 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309696 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309699 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309702 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309705 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309708 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309710 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309713 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309715 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309718 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309721 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309723 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309726 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309728 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309731 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.311565 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309733 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309736 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309739 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309742 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309744 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309747 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309750 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309752 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309755 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309757 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309760 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309763 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309765 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309768 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309770 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309773 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309775 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309778 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309780 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309782 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.312041 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309786 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309788 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309791 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309793 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309796 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309799 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309802 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309804 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309807 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309809 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309812 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309814 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.309817 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310374 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310381 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310385 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310388 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310391 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310393 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310396 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.312522 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310399 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310402 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310405 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310407 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310410 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310413 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310415 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310418 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310421 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310423 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310426 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310429 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310433 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310436 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310438 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310441 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310444 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310446 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310449 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.313016 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310451 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310454 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310457 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310459 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310462 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310464 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310467 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310470 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310472 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310475 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310477 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310480 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310482 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310487 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310491 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310494 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310497 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310500 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310503 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.313467 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310506 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310508 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310510 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310513 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310516 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310519 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310521 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310524 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310527 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310529 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310532 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310534 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310537 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310539 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310542 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310545 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310547 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310550 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310552 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310555 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.313939 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310558 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310560 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310563 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310565 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310568 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310571 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310573 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310576 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310579 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310581 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310584 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310586 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310589 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310591 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310594 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310596 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310599 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310602 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310605 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310607 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.314419 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310611 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310614 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310616 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310619 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310621 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310624 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310627 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310630 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310632 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310635 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310638 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310641 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310645 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310649 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310653 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310671 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310676 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310680 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310684 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310689 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.314968 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.310692 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310805 2587 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310815 2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310823 2587 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310827 2587 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310832 2587 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310836 2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310840 2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310845 2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310848 2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310852 2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310855 2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310859 2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310862 2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310865 2587 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310868 2587 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310871 2587 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310874 2587 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310877 2587 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310882 2587 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310885 2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310888 2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310892 2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310895 2587 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:41:41.315437 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310911 2587 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310915 2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310918 2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310922 2587 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310925 2587 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310928 2587 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310931 2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310934 2587 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310937 2587 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310942 2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310946 2587 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310949 2587 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310952 2587 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310955 2587 flags.go:64] FLAG: --enable-server="true" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310959 2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310963 2587 flags.go:64] FLAG: --event-burst="100" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310967 2587 flags.go:64] FLAG: --event-qps="50" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310970 2587 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310974 2587 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310977 2587 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310982 2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310985 2587 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310988 2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310991 2587 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310994 2587 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:41:41.316044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.310998 2587 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311001 2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311004 2587 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311007 2587 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311010 2587 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311013 2587 flags.go:64] FLAG: --feature-gates="" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311017 2587 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311020 2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311024 2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311027 2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311030 2587 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311033 2587 flags.go:64] FLAG: --help="false" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311037 2587 flags.go:64] FLAG: --hostname-override="ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311040 2587 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311044 2587 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311047 2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311050 2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311054 2587 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311057 2587 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311060 2587 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311063 2587 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311069 2587 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311072 2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311076 2587 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:41:41.316747 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311081 2587 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311084 2587 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311087 2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311090 2587 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311093 2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311096 2587 flags.go:64] FLAG: --lock-file="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311100 2587 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311103 2587 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311106 2587 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311112 2587 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311115 2587 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311118 2587 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311121 2587 flags.go:64] FLAG: --logging-format="text" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311124 2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311128 2587 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311131 2587 flags.go:64] FLAG: --manifest-url="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311134 2587 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311139 2587 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311142 2587 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311145 2587 flags.go:64] FLAG: --max-pods="110" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311149 2587 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311152 2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311155 2587 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311158 2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311161 2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:41:41.317349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311164 2587 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311167 2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311175 2587 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311178 2587 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311183 2587 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311186 2587 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311189 2587 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311193 2587 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311197 2587 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311200 2587 flags.go:64] FLAG: --port="10250" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311203 2587 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311206 2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-083fd340b6ba3b1eb" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311209 2587 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311212 2587 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311215 2587 flags.go:64] FLAG: --register-node="true" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311218 2587 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311222 2587 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311225 2587 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311228 2587 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311231 2587 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311235 2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311238 2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311241 2587 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311244 2587 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311247 2587 flags.go:64] FLAG: --runonce="false" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311250 2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:41:41.317993 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311253 2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311256 2587 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311259 2587 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311262 2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311266 2587 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311269 2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311272 2587 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311275 2587 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311278 2587 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311282 2587 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311286 2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311289 2587 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311292 2587 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311297 2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311303 2587 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311306 2587 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311308 2587 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311314 2587 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311317 2587 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311320 2587 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311323 2587 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311326 2587 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311329 2587 flags.go:64] FLAG: --v="2" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311334 2587 flags.go:64] FLAG: --version="false" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311338 2587 flags.go:64] FLAG: --vmodule="" Oct 29 21:41:41.318951 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311342 2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311346 2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311486 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311490 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311494 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311497 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311500 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311503 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311505 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311508 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311510 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311513 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311516 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311518 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311521 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311523 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311526 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311531 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311533 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311536 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.319732 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311540 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311542 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311545 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311548 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311551 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311554 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311556 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311560 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311565 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311568 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311571 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311574 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311577 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311580 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311583 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311585 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311588 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311590 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311593 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.320354 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311597 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311600 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311604 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311607 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311610 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311613 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311615 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311618 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311620 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311623 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311627 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311633 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311636 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311640 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311642 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311645 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311648 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311651 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311654 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311656 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.320804 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311659 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311662 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311664 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311667 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311669 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311672 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311675 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311677 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311680 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311682 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311685 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311688 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311690 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311693 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311695 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311698 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311700 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311703 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311705 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311708 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.321299 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311711 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311713 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311717 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311720 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311723 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311727 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311730 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311732 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311735 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311737 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311740 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311743 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311745 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311748 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311751 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311753 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311756 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311758 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311761 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311763 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.321766 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311766 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311768 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311771 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311773 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311776 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311778 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311781 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311783 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311786 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.311812 2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311851 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311855 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311858 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311861 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311864 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311868 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.322258 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311871 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311875 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311878 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311881 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311884 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311886 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311889 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311891 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311894 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311897 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311899 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311913 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311916 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311919 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311921 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311924 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311926 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311929 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311932 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.322653 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311935 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311937 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311940 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311942 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311945 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311947 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311950 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311952 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311955 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311958 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311960 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311963 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311967 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311969 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311974 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311977 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311980 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311983 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311985 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.323113 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311988 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311991 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311993 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.311996 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312000 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312004 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312007 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312010 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312013 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312016 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312018 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312021 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312023 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312026 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312028 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312031 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312035 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312039 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312042 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.323555 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312045 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312047 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312050 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312053 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312056 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312059 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312062 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312065 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312068 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312071 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312074 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312076 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312079 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312081 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312084 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312086 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312089 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312093 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312095 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312098 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.324045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312100 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312103 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312106 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312108 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312111 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312113 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312116 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312118 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312121 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312123 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312126 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312129 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312131 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312133 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312136 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312138 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312141 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312143 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312146 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312150 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.324559 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312153 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312155 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.312158 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.320760 2587 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.320778 2587 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.320793 2587 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.320799 2587 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320898 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320919 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320923 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320926 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320929 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320932 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320936 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320938 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320941 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320944 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320947 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320950 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320952 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.325093 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320955 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320957 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320960 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320962 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320965 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320968 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320971 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320974 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320976 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320984 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320987 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320990 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320993 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320996 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.320999 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321001 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321004 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321006 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.325582 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321011 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321015 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321018 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321021 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321024 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321027 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321029 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321032 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321035 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321037 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321040 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321043 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321046 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321049 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321051 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321055 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321058 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321060 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321063 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321065 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.326067 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321069 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321071 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321074 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321076 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321079 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321082 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321085 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321088 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321091 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321093 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321096 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321099 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321101 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321104 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321107 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321109 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321112 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321114 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321117 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.326636 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321119 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321123 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321127 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321129 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321132 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321135 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321137 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321140 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321143 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321146 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321149 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321152 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321155 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321158 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321160 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321163 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321165 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321168 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321171 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.327107 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321174 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321177 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321179 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321182 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321185 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321187 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321190 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321192 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321195 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321198 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321200 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321202 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321205 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321208 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321210 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321213 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.327597 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321216 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.321243 2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321388 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321393 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321397 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321400 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321403 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321406 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321410 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321412 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321415 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321417 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321420 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321423 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321426 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321428 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.327991 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321431 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321434 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321437 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321439 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321442 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321444 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321447 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321449 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321452 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321455 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321457 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321460 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321462 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321465 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321467 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321470 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321472 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321475 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321478 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321480 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.328516 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321483 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321485 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321488 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321490 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321494 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321496 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321499 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321501 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321504 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321507 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321510 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321513 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321516 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321518 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321521 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321523 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321526 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321528 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321531 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321533 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.329337 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321536 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321538 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321541 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321543 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321547 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321551 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321554 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321556 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321559 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321562 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321565 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321568 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321571 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321573 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321575 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321578 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321581 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321583 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321586 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.329839 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321589 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321591 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321594 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321597 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321599 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321602 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321605 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321607 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321610 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321612 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321615 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321617 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321620 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321622 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321625 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321628 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321630 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321633 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321635 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.330414 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321638 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321640 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321643 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321645 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321648 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321650 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321653 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321655 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321658 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321660 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321663 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321665 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321668 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321672 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.321695 2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321728 2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:41.330859 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321733 2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321737 2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321740 2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321743 2587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321746 2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321748 2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321751 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321754 2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321757 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321759 2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321762 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321764 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321767 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321770 2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321773 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321776 2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321778 2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321781 2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321784 2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321786 2587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:41.331423 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321789 2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321791 2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321794 2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321797 2587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321799 2587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321802 2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321804 2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321807 2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321810 2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321813 2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321815 2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321818 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321821 2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321826 2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321829 2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321831 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321834 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321836 2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321840 2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321843 2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:41.332045 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321846 2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321849 2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321852 2587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321855 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321857 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321860 2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321862 2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321865 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321867 2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321870 2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321872 2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321875 2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321877 2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321880 2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321882 2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321885 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321887 2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321890 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321892 2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:41.332513 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321895 2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321897 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321913 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321921 2587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321925 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321927 2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321931 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321934 2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321937 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321939 2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321943 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321946 2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321948 2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321951 2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321953 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321956 2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321959 2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321961 2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321964 2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321967 2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:41.333033 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321969 2587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321972 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321975 2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321977 2587 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321980 2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321982 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321986 2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321990 2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321993 2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321996 2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.321998 2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322001 2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322004 2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322007 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322010 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322014 2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322017 2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322021 2587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322025 2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:41.333547 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322028 2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322030 2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322033 2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322035 2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322038 2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322041 2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:41.322043 2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.322103 2587 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.324436 2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.325510 2587 server.go:1014] "Starting client certificate rotation" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.325625 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.326464 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.327810 2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:41.334245 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.328999 2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:41.341594 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.341576 2587 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:41:41.346430 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.346415 2587 log.go:25] "Validated CRI v1 image API" Oct 29 21:41:41.347294 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.347271 2587 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:41:41.352106 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.352087 2587 fs.go:135] Filesystem UUIDs: map[26e1c196-46c3-457d-b7f0-fe6fde7ebb88:/dev/nvme0n1p3 6937126b-5e4a-4f6e-b0b3-cf619608a308:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 29 21:41:41.352190 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.352105 2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 21:41:41.359860 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.359743 2587 manager.go:217] Machine: {Timestamp:2026-10-29 21:41:41.358282489 +0000 UTC m=+0.328495298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099740 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f1ac009482ea940e546ef9ff9381 SystemUUID:ec29f1ac-0094-82ea-940e-546ef9ff9381 BootID:9070598f-f861-448a-9134-9b6317d2a467 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:6c:42:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:6c:42:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:07:3f:90:0c:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:41:41.360593 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.360583 2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:41:41.360676 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.360665 2587 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:41:41.364627 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.364569 2587 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:41:41.365052 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.364634 2587 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:41:41.365052 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.364985 2587 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:41:41.365052 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.364998 2587 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:41:41.365214 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.365131 2587 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:41.365384 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.365369 2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:41.366619 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.366607 2587 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:41:41.370959 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.370947 2587 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:41:41.371075 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371065 2587 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:41:41.371851 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371842 2587 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:41:41.371887 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371856 2587 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:41:41.371887 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371866 2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:41:41.371887 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371875 2587 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:41:41.371887 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.371883 2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:41:41.372238 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.372224 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:41.374774 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.374755 2587 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:41:41.381669 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.381657 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:41.382670 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382658 2587 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:41:41.382709 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382688 2587 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:41:41.382855 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382845 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:41:41.382896 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382861 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:41:41.382896 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382867 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:41:41.382896 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382873 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:41:41.382896 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382878 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:41:41.382896 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.382884 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:41:41.383723 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383715 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:41:41.383757 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383728 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:41:41.383757 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383740 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:41:41.383757 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383746 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:41:41.383757 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383754 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:41:41.383922 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.383765 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:41:41.385349 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.385340 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:41:41.385390 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.385349 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:41:41.389127 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.389112 2587 server.go:1263] "Started kubelet" Oct 29 21:41:41.389254 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.389222 2587 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:41:41.389339 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.389229 2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:41:41.389409 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.389394 2587 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:41:41.389866 ip-10-0-141-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:41:41.390828 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.390811 2587 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:41:41.392551 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.392537 2587 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:41:41.395649 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.395630 2587 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 21:41:41.398004 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.397991 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:41.398680 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.398664 2587 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:41:41.398784 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.398764 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.399433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.399329 2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:41:41.399433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.399342 2587 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:41:41.399433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.399353 2587 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:41:41.399433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.399411 2587 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:41:41.399433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.399421 2587 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:41:41.401791 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.401775 2587 factory.go:55] Registering systemd factory Oct 29 21:41:41.401791 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.401791 2587 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:41:41.402009 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.401997 2587 factory.go:153] Registering CRI-O factory Oct 29 21:41:41.402060 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.402012 2587 factory.go:223] Registration of the crio container factory successfully Oct 29 21:41:41.402105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.402060 2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:41:41.402105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.402080 2587 factory.go:103] Registering Raw factory Oct 29 21:41:41.402105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.402090 2587 manager.go:1199] Started watching for new ooms in manager Oct 29 21:41:41.403209 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.403196 2587 manager.go:319] Starting recovery of all containers Oct 29 21:41:41.413725 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.413704 2587 manager.go:324] Recovery completed Oct 29 21:41:41.418029 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.418014 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.418288 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.418272 2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 21:41:41.419411 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.418269 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-100.ec2.internal.18e31e092d6a8093 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:41.389082771 +0000 UTC m=+0.359295580,LastTimestamp:2026-10-29 21:41:41.389082771 +0000 UTC m=+0.359295580,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-100.ec2.internal,}" Oct 29 21:41:41.420397 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420379 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.420480 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420408 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.420480 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420419 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.420937 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420919 2587 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:41:41.420937 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420935 2587 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:41:41.421033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420953 2587 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:41:41.421033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.420924 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.423537 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423500 2587 policy_none.go:50] "Start" Oct 29 21:41:41.423537 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423517 2587 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:41:41.423537 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423530 2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:41:41.423929 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423897 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.423978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423949 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.423978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.423966 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.425591 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.425578 2587 policy_none.go:44] "Start" Oct 29 21:41:41.428879 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.428852 2587 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 21:41:41.437113 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.437095 2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:41.439084 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.439062 2587 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 21:41:41.439160 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.439080 2587 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 21:41:41.439211 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.439108 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-100.ec2.internal.18e31e092f484957 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:41.420394839 +0000 UTC m=+0.390607648,LastTimestamp:2026-10-29 21:41:41.420394839 +0000 UTC m=+0.390607648,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-100.ec2.internal,}" Oct 29 21:41:41.444618 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.444597 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sr5kx" Oct 29 21:41:41.448476 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.448451 2587 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:41:41.451292 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451276 2587 manager.go:341] "Starting Device Plugin manager" Oct 29 21:41:41.451365 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.451307 2587 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:41:41.451365 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451317 2587 server.go:87] "Starting device plugin registration server" Oct 29 21:41:41.451562 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451545 2587 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:41:41.451640 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451561 2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:41:41.451750 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451735 2587 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:41:41.451857 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451812 2587 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:41:41.451857 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.451820 2587 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:41:41.452260 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.452223 2587 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:41:41.452353 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.452290 2587 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.459463 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.459440 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sr5kx" Oct 29 21:41:41.470433 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.470349 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-100.ec2.internal.18e31e092f489204 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-100.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:41.420413444 +0000 UTC m=+0.390626252,LastTimestamp:2026-10-29 21:41:41.420413444 +0000 UTC m=+0.390626252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-100.ec2.internal,}" Oct 29 21:41:41.470543 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.470522 2587 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 21:41:41.551674 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.551645 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.555076 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.555017 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.555076 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.555048 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.555076 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.555060 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.555235 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.555093 2587 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.575969 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.575941 2587 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.575969 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.575971 2587 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-100.ec2.internal\": node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.631417 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.631245 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.658203 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.658160 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:41:41.659453 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.659430 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:41:41.659561 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.659460 2587 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:41:41.659561 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.659484 2587 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:41:41.659561 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.659525 2587 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 21:41:41.679654 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.679631 2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:41.731696 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.731662 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.759849 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.759819 2587 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal"] Oct 29 21:41:41.759945 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.759917 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.760987 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.760973 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.761043 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.761002 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.761043 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.761014 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.763037 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763025 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.763195 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763180 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.763235 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763210 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.763843 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763816 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.763978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763852 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.763978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763864 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.763978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763917 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.763978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763941 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.763978 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.763951 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.765822 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.765795 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.765822 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.765822 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:41.766749 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.766733 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:41.766819 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.766764 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:41.766819 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.766780 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:41.785018 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.785000 2587 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-100.ec2.internal\" not found" node="ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.788021 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.788005 2587 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-100.ec2.internal\" not found" node="ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.801433 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.801417 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.801493 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.801440 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.801493 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.801461 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.832453 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.832386 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:41.902323 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902294 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.902397 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902327 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.902397 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902342 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.902465 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902413 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.902465 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902419 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.902523 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:41.902457 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:41.933291 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:41.933267 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.033972 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.033941 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.088163 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.088104 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:42.091205 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.091192 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:42.134746 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.134715 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.237124 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.237071 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.326170 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.326139 2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 21:41:42.326714 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.326303 2587 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 21:41:42.337659 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.337634 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.438882 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.438800 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.465675 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.465617 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:42 +0000 UTC" deadline="2028-07-17 21:01:04.716296642 +0000 UTC" Oct 29 21:41:42.465675 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.465665 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15047h19m22.250636792s" Oct 29 21:41:42.539771 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.539733 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.623011 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.622979 2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:42.626072 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.625281 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:42.626072 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.625316 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:42.626072 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.625330 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:42.630211 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.630193 2587 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:42.640490 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.640466 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.691030 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:42.690993 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97accf019645b7962e49aedeb58f438f.slice/crio-c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352 WatchSource:0}: Error finding container c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352: Status 404 returned error can't find the container with id c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352 Oct 29 21:41:42.693068 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:42.693033 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9c0102f0b78e22ae4ea8593b59d168.slice/crio-ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339 WatchSource:0}: Error finding container ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339: Status 404 returned error can't find the container with id ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339 Oct 29 21:41:42.697805 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.697786 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:41:42.740876 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:42.740842 2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:41:42.804522 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.804494 2587 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:42.901106 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.901060 2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:41:42.926038 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.926005 2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:42.927195 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.927179 2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:41:42.951686 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.951667 2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:42.970492 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:42.970462 2587 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:43.176048 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.176015 2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:43.373106 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.373020 2587 apiserver.go:52] "Watching apiserver" Oct 29 21:41:43.399975 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.399946 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:43.406193 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.406162 2587 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:41:43.406662 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.406635 2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zs2sf","kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qlm4k","openshift-dns/node-resolver-2cmw5","openshift-multus/multus-additional-cni-plugins-fgvh6","openshift-multus/multus-xbs4b","openshift-multus/network-metrics-daemon-z8gwh","kube-system/konnectivity-agent-qjxpk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl","openshift-image-registry/node-ca-gstvd","openshift-network-operator/iptables-alerter-7xwz2","kube-system/global-pull-secret-syncer-z6ll7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal","openshift-network-diagnostics/network-check-target-snzg7"] Oct 29 21:41:43.409378 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.409330 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.409498 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.409410 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:43.411701 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.411680 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.414540 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.414064 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.416458 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.416434 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.419313 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.419295 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.421972 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.421929 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.424280 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.424259 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.429558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.429524 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:43.429787 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.429606 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:43.430066 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.430043 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:41:43.431347 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.431327 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:41:43.431528 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.431515 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:41:43.431659 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.431647 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.431746 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.431734 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.431876 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.431864 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.432060 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.432045 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.433846 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.433112 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.433846 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.433283 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.436412 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.436396 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.439050 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.439026 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.441729 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.441699 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.441827 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.441774 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:43.459773 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.459743 2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:43.459890 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.459845 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lk5nz" Oct 29 21:41:43.472839 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.472813 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lk5nz" Oct 29 21:41:43.501556 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.501530 2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:41:43.511793 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511760 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511806 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511835 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511863 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511893 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511920 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511965 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.511989 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512015 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512039 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512068 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512065 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccm7\" (UniqueName: \"kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512091 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512116 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512144 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512172 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512196 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512218 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512250 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512274 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512298 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512322 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512372 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512392 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512409 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512429 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512443 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512457 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.512547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512472 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512488 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482h2\" (UniqueName: \"kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512504 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512536 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mrb\" (UniqueName: \"kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512553 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512568 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512704 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512733 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512784 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512822 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512866 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512938 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.512975 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513002 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513030 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513068 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.513386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513114 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513138 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513157 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513199 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513221 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513286 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513319 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513350 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhbl\" (UniqueName: \"kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513378 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9fp\" (UniqueName: \"kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513411 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513430 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513460 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513486 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513520 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513561 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513598 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513639 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513683 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513725 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfktv\" (UniqueName: \"kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513757 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513785 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513873 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513931 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.513963 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514008 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514033 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514061 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514084 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514125 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514151 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514185 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514207 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n896\" (UniqueName: \"kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514228 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.514970 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514247 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjs5\" (UniqueName: \"kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.515901 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514263 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.515901 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514280 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.515901 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514340 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.515901 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.514357 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzw7g\" (UniqueName: \"kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.539249 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.538448 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:41:43.539249 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.538981 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:41:43.539513 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539263 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.539513 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539371 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:41:43.539513 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539430 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.539513 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539380 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:41:43.539718 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539556 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:41:43.539718 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539675 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:41:43.539718 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539688 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:41:43.539718 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539701 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:41:43.539919 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539715 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:41:43.539919 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539716 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:41:43.539919 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539850 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.539919 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539880 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:41:43.540208 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539921 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:41:43.540208 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.539937 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.540208 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540031 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.540208 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540040 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540252 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540262 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540275 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540286 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540296 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540340 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:41:43.540394 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540371 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:41:43.540671 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540429 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:41:43.540671 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540463 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:41:43.540671 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.540503 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:41:43.615009 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.614943 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfktv\" (UniqueName: \"kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615026 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615052 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615074 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615092 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615112 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615134 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615160 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615188 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.615212 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615211 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615233 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615267 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615286 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615303 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n896\" (UniqueName: \"kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615319 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615340 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjs5\" (UniqueName: \"kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615357 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615374 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615394 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615400 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615433 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzw7g\" (UniqueName: \"kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615470 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615495 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615511 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615531 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615558 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.615734 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615579 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615597 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615626 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615648 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615665 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615690 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccm7\" (UniqueName: \"kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615719 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615735 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615743 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615792 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615818 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615837 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615844 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615870 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615905 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615930 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615959 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615996 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.616603 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616017 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616041 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616064 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616090 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616129 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616157 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616192 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616216 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482h2\" (UniqueName: \"kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616233 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616252 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mrb\" (UniqueName: \"kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616281 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616308 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616335 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616362 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616390 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616416 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616439 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.617558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616442 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616474 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616501 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616529 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616555 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616580 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616609 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616638 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616667 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616695 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616724 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616728 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616769 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616799 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616831 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhbl\" (UniqueName: \"kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616868 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9fp\" (UniqueName: \"kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616897 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.618435 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.616926 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617001 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617007 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617089 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617122 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617149 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617178 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617219 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617244 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617293 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617349 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617809 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617880 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617249 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617939 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.617944 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618044 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618184 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.619315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618280 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618303 2587 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618324 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618363 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618480 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618651 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618709 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618807 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.618961 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619021 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619059 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619096 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619135 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619351 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619408 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619508 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.615796 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619558 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.620244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619593 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619631 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.619667 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620065 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620149 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620182 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620194 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620243 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620308 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620359 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.621177 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.620701 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.628204 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.627844 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.628204 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.628126 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.628473 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.628238 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.629132 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629103 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.629321 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629204 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.629537 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629511 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.629757 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629645 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.629873 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629850 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.629873 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.629836 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.630151 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.630083 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.630226 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.630181 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:44.130155888 +0000 UTC m=+3.100368707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.630289 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630266 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.630502 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630419 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:43.631105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630569 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.631105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630709 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.631105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630787 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.631105 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.630871 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.631357 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631179 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.631357 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631212 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.631462 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.631360 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.631462 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.631420 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:44.131396181 +0000 UTC m=+3.101608994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.631570 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631479 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.631570 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631550 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.631667 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631453 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.631667 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631639 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.631854 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.631769 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.632659 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.632515 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.633215 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.633071 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.636287 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.634430 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.636480 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.636454 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.645894 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.645858 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjs5\" (UniqueName: \"kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.645894 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.645884 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n896\" (UniqueName: \"kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.646401 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.646358 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzw7g\" (UniqueName: \"kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.646856 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.646830 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccm7\" (UniqueName: \"kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.648388 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.648366 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mrb\" (UniqueName: \"kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.648672 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.648648 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfktv\" (UniqueName: \"kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.654110 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.654087 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:43.654110 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.654112 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:43.654282 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.654128 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.654282 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:43.654212 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:41:44.154190273 +0000 UTC m=+3.124403075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.657166 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.657138 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482h2\" (UniqueName: \"kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.657399 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.657380 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhbl\" (UniqueName: \"kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:43.658464 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.658420 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9fp\" (UniqueName: \"kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.665963 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.665872 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerStarted","Data":"c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352"} Oct 29 21:41:43.667491 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.667464 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" event={"ID":"4c9c0102f0b78e22ae4ea8593b59d168","Type":"ContainerStarted","Data":"ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339"} Oct 29 21:41:43.729041 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.728999 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:41:43.736156 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.736116 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:41:43.741349 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.741317 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cb51a1_45ef_4b15_b6f7_74cf138af237.slice/crio-f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8 WatchSource:0}: Error finding container f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8: Status 404 returned error can't find the container with id f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8 Oct 29 21:41:43.747558 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.747526 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9885f82f_5fef_44bd_84b4_de771bc1e0e7.slice/crio-13f42a2bf7271f09c89cf4f40991f462b7efc045981d02ed167c3d76c68372c7 WatchSource:0}: Error finding container 13f42a2bf7271f09c89cf4f40991f462b7efc045981d02ed167c3d76c68372c7: Status 404 returned error can't find the container with id 13f42a2bf7271f09c89cf4f40991f462b7efc045981d02ed167c3d76c68372c7 Oct 29 21:41:43.748288 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.748258 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:41:43.761798 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.761763 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57d5263_1744_42ca_ae20_2c52ab13ba81.slice/crio-a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309 WatchSource:0}: Error finding container a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309: Status 404 returned error can't find the container with id a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309 Oct 29 21:41:43.766737 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.766704 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbs4b" Oct 29 21:41:43.771291 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.771265 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:41:43.779911 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.778939 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:41:43.779911 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.779069 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e456b9_c486_4c7b_8339_a87159a8fe7e.slice/crio-2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d WatchSource:0}: Error finding container 2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d: Status 404 returned error can't find the container with id 2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d Oct 29 21:41:43.782804 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.782768 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:43.785442 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.785409 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8f95dd_9d27_4626_891b_e5ef541d7562.slice/crio-a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04 WatchSource:0}: Error finding container a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04: Status 404 returned error can't find the container with id a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04 Oct 29 21:41:43.790550 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.790525 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf7bd76_73c6_4cf0_bad9_f7830153f335.slice/crio-43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d WatchSource:0}: Error finding container 43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d: Status 404 returned error can't find the container with id 43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d Oct 29 21:41:43.791738 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.791721 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:41:43.798562 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.798526 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b0e867_b455_45f3_94cf_3aeae43cb6ee.slice/crio-62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74 WatchSource:0}: Error finding container 62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74: Status 404 returned error can't find the container with id 62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74 Oct 29 21:41:43.804999 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.804951 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295393c7_fd50_401c_a65b_95293cc4953a.slice/crio-264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c WatchSource:0}: Error finding container 264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c: Status 404 returned error can't find the container with id 264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c Oct 29 21:41:43.813560 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:43.813536 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:41:43.825124 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:41:43.825087 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb85288_ba12_4aa5_aa3a_33059623a339.slice/crio-7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6 WatchSource:0}: Error finding container 7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6: Status 404 returned error can't find the container with id 7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6 Oct 29 21:41:44.225156 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.224383 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:44.225156 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.224445 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:44.225156 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.224478 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:44.225156 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225143 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:44.225596 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225201 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:44.225596 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225231 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:44.225596 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225323 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:41:45.225288551 +0000 UTC m=+4.195501365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:44.225859 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225830 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:44.225943 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225924 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:45.225901207 +0000 UTC m=+4.196114023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:44.225943 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.225921 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:44.226063 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.226000 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:45.225969076 +0000 UTC m=+4.196181900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:44.476143 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.476039 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:44 +0000 UTC" deadline="2028-07-16 00:11:15.835134944 +0000 UTC" Oct 29 21:41:44.476143 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.476088 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15002h29m31.35904918s" Oct 29 21:41:44.660646 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.660605 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:44.660983 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.660929 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:44.661562 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.661527 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:44.661706 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:44.661635 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:44.682414 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.682372 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gstvd" event={"ID":"295393c7-fd50-401c-a65b-95293cc4953a","Type":"ContainerStarted","Data":"264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c"} Oct 29 21:41:44.693669 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.693625 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" event={"ID":"fbf7bd76-73c6-4cf0-bad9-f7830153f335","Type":"ContainerStarted","Data":"43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d"} Oct 29 21:41:44.699556 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.699512 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" event={"ID":"9885f82f-5fef-44bd-84b4-de771bc1e0e7","Type":"ContainerStarted","Data":"13f42a2bf7271f09c89cf4f40991f462b7efc045981d02ed167c3d76c68372c7"} Oct 29 21:41:44.709171 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.708486 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" event={"ID":"4c9c0102f0b78e22ae4ea8593b59d168","Type":"ContainerStarted","Data":"eca6484930ff55305de3a36d4f69327d74e7346f4b8fb42981121cfc13ac2f8c"} Oct 29 21:41:44.714337 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.714299 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6"} Oct 29 21:41:44.721809 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.721766 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjxpk" event={"ID":"a3b0e867-b455-45f3-94cf-3aeae43cb6ee","Type":"ContainerStarted","Data":"62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74"} Oct 29 21:41:44.723944 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.723907 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerStarted","Data":"a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04"} Oct 29 21:41:44.727577 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.727500 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbs4b" event={"ID":"29e456b9-c486-4c7b-8339-a87159a8fe7e","Type":"ContainerStarted","Data":"2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d"} Oct 29 21:41:44.730025 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.729986 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2cmw5" event={"ID":"b57d5263-1744-42ca-ae20-2c52ab13ba81","Type":"ContainerStarted","Data":"a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309"} Oct 29 21:41:44.737227 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.737188 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xwz2" event={"ID":"11cb51a1-45ef-4b15-b6f7-74cf138af237","Type":"ContainerStarted","Data":"f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8"} Oct 29 21:41:44.752667 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:44.752592 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" podStartSLOduration=1.7525681400000002 podStartE2EDuration="1.75256814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.751463521 +0000 UTC m=+3.721676343" watchObservedRunningTime="2026-10-29 21:41:44.75256814 +0000 UTC m=+3.722780963" Oct 29 21:41:45.235570 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.235280 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:45.235722 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.235602 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:45.235722 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.235641 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:45.235896 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.235878 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:45.235960 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.235905 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:45.235960 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.235918 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:45.236106 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.236004 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.235969351 +0000 UTC m=+6.206182176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:45.236177 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.236105 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:45.236177 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.236151 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.236135626 +0000 UTC m=+6.206348439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:45.236286 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.236218 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:45.236286 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.236252 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.236241313 +0000 UTC m=+6.206454113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:45.477091 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.477034 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:44 +0000 UTC" deadline="2028-06-14 22:41:17.459837077 +0000 UTC" Oct 29 21:41:45.477091 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.477088 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14256h59m31.98275424s" Oct 29 21:41:45.661187 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.661156 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:45.661394 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:45.661318 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:45.759132 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.759037 2587 generic.go:356] "Generic (PLEG): container finished" podID="97accf019645b7962e49aedeb58f438f" containerID="ef6e0238c91f1a72a1cf81ff98597419bd76bfcf3e257d7a5dcd04174b43a1ba" exitCode=0 Oct 29 21:41:45.760199 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:45.760169 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerDied","Data":"ef6e0238c91f1a72a1cf81ff98597419bd76bfcf3e257d7a5dcd04174b43a1ba"} Oct 29 21:41:46.659765 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:46.659724 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:46.660303 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:46.659876 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:46.660303 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:46.659981 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:46.660303 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:46.660060 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:46.769149 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:46.768940 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerStarted","Data":"f85d3b5f5b0051cab9d66ac0ff18e9904d22f45565538fd3a0b1b85fb30b4457"} Oct 29 21:41:47.254391 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.254349 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:47.254391 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.254395 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:47.254657 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.254425 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:47.254657 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.254600 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:47.254657 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.254619 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:47.254657 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.254632 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:47.254880 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.254686 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.254672917 +0000 UTC m=+10.224885729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:47.255134 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.255079 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:47.255134 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.255124 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.255112776 +0000 UTC m=+10.225325587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:47.255305 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.255163 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:47.255305 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.255185 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.255177971 +0000 UTC m=+10.225390782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:47.662840 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.662211 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:47.662840 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:47.662376 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:47.786602 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.786558 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="c0473c44e6e5b0edf59daaabc19ed3e3361a5f982954e885ffd86335b59d44f4" exitCode=0 Oct 29 21:41:47.787811 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.787780 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"c0473c44e6e5b0edf59daaabc19ed3e3361a5f982954e885ffd86335b59d44f4"} Oct 29 21:41:47.867443 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:47.866625 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" podStartSLOduration=4.866604119 podStartE2EDuration="4.866604119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:46.823756432 +0000 UTC m=+5.793969245" watchObservedRunningTime="2026-10-29 21:41:47.866604119 +0000 UTC m=+6.836816943" Oct 29 21:41:48.660033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:48.660001 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:48.660235 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:48.660145 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:48.660545 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:48.660528 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:48.660645 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:48.660597 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:48.792386 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:48.792344 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjxpk" event={"ID":"a3b0e867-b455-45f3-94cf-3aeae43cb6ee","Type":"ContainerStarted","Data":"ee46a8d4580e1990ebb1ada31b32198f92e5a5d267ffb4dc8990bf9148dbf33c"} Oct 29 21:41:49.660731 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:49.660684 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:49.660932 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:49.660801 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:49.800605 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:49.800556 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gstvd" event={"ID":"295393c7-fd50-401c-a65b-95293cc4953a","Type":"ContainerStarted","Data":"60602254f52974c4cda61034345e44b450218e920cecead8f4907160b6a2dfca"} Oct 29 21:41:49.846898 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:49.846260 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qjxpk" podStartSLOduration=3.757225735 podStartE2EDuration="7.846237582s" totalImagesPullingTime="4.089011847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:48.838700733 +0000 UTC m=+7.808913556" watchObservedRunningTime="2026-10-29 21:41:49.846237582 +0000 UTC m=+8.816450466" Oct 29 21:41:50.177356 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.177263 2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:50.178328 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.178307 2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:41:50.223639 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.223565 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gstvd" podStartSLOduration=2.888444391 podStartE2EDuration="8.223544323s" totalImagesPullingTime="5.335099932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:49.84682647 +0000 UTC m=+8.817039292" watchObservedRunningTime="2026-10-29 21:41:50.223544323 +0000 UTC m=+9.193757150" Oct 29 21:41:50.661450 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.660633 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:50.661450 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:50.660777 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:50.661450 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.661273 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:50.661450 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:50.661387 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:50.808864 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:50.808345 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" event={"ID":"fbf7bd76-73c6-4cf0-bad9-f7830153f335","Type":"ContainerStarted","Data":"41e31c8f70d03fb4c7cb7b3b4bd811a9f68c38c24b1d1f9236aa57177fff9fd2"} Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.295226 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.295279 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.295306 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295480 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295498 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295510 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295568 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:41:59.295552089 +0000 UTC m=+18.265764891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295642 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295671 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:59.295662038 +0000 UTC m=+18.265874836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295724 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:51.296128 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.295750 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:59.295741896 +0000 UTC m=+18.265954692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:51.633184 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.632934 2587 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:41:51.664170 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.661650 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:51.664170 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:51.661823 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:51.815841 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.815767 2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:41:51.816350 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:51.816298 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" event={"ID":"fbf7bd76-73c6-4cf0-bad9-f7830153f335","Type":"ContainerStarted","Data":"a1ce824a0c6638b1d9ee2df3682cad5b6f680fc8add13b86b4073f2a7379eaf0"} Oct 29 21:41:52.459478 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.459194 2587 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:41:51.632970751Z","UUID":"a28c7d29-7c93-458c-98b7-270343fc9c94","Handler":null,"Name":"","Endpoint":""} Oct 29 21:41:52.460763 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.460739 2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:41:52.460763 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.460770 2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:41:52.660860 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.660826 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:52.661056 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:52.660964 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:52.661056 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.661020 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:52.661163 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:52.661105 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:52.820278 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.820173 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="e732f391bdead3fda4af7d5b2e14cde677ac1c0f94f985832f7b298690cd95df" exitCode=0 Oct 29 21:41:52.820278 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.820264 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"e732f391bdead3fda4af7d5b2e14cde677ac1c0f94f985832f7b298690cd95df"} Oct 29 21:41:52.822359 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.822322 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbs4b" event={"ID":"29e456b9-c486-4c7b-8339-a87159a8fe7e","Type":"ContainerStarted","Data":"17e8321c162857f67b3f49f42a93153b24b8a7d058326ec13021fdea99d3d188"} Oct 29 21:41:52.824114 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.824091 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2cmw5" event={"ID":"b57d5263-1744-42ca-ae20-2c52ab13ba81","Type":"ContainerStarted","Data":"8436a29ee040a0e97ca06b28e36e27f587e124e44d72d8f807e1461d98a748cd"} Oct 29 21:41:52.912987 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.912933 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xbs4b" podStartSLOduration=2.65968704 podStartE2EDuration="10.912898254s" totalImagesPullingTime="8.253211214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:52.912320696 +0000 UTC m=+11.882533515" watchObservedRunningTime="2026-10-29 21:41:52.912898254 +0000 UTC m=+11.883111076" Oct 29 21:41:52.953189 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:52.953131 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2cmw5" podStartSLOduration=2.503485688 podStartE2EDuration="10.95311453s" totalImagesPullingTime="8.449628842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:52.953009785 +0000 UTC m=+11.923222607" watchObservedRunningTime="2026-10-29 21:41:52.95311453 +0000 UTC m=+11.923327348" Oct 29 21:41:53.661679 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:53.661120 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:53.661679 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:53.661280 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:53.828917 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:53.828863 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" event={"ID":"fbf7bd76-73c6-4cf0-bad9-f7830153f335","Type":"ContainerStarted","Data":"cf73ce2aa7826ceb7be3d24aca96d8de59604c491ecfc7b83cf42503305851fd"} Oct 29 21:41:53.875635 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:53.874246 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" podStartSLOduration=3.388888529 podStartE2EDuration="11.874229785s" totalImagesPullingTime="8.485341256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:53.874217867 +0000 UTC m=+12.844430688" watchObservedRunningTime="2026-10-29 21:41:53.874229785 +0000 UTC m=+12.844442604" Oct 29 21:41:54.659744 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.659710 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:54.659882 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.659763 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:54.659989 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:54.659919 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:54.659989 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:54.659976 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:54.832397 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.832357 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" event={"ID":"9885f82f-5fef-44bd-84b4-de771bc1e0e7","Type":"ContainerStarted","Data":"ecf2f119da7988164df5bd527c775821738fc83176c7f912ddc7b8c67afe7b70"} Oct 29 21:41:54.835329 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.835290 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="87a4c1bc944f5957b8ec8ec0d038df2b06692555030d3526f304fa26aef02717" exitCode=0 Oct 29 21:41:54.835478 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.835360 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"87a4c1bc944f5957b8ec8ec0d038df2b06692555030d3526f304fa26aef02717"} Oct 29 21:41:54.837397 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.837360 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xwz2" event={"ID":"11cb51a1-45ef-4b15-b6f7-74cf138af237","Type":"ContainerStarted","Data":"0de87db809dfac50f2a13d3f2cc0ad4ceab6882d700cbc3dac61e2d149303e94"} Oct 29 21:41:54.876933 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.876847 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" podStartSLOduration=2.003789169 podStartE2EDuration="12.876835931s" totalImagesPullingTime="10.873046762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:54.876582375 +0000 UTC m=+13.846795192" watchObservedRunningTime="2026-10-29 21:41:54.876835931 +0000 UTC m=+13.847048749" Oct 29 21:41:54.966540 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:54.966490 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xwz2" podStartSLOduration=4.11159138 podStartE2EDuration="12.966475821s" totalImagesPullingTime="8.854884441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:54.965949361 +0000 UTC m=+13.936162179" watchObservedRunningTime="2026-10-29 21:41:54.966475821 +0000 UTC m=+13.936688639" Oct 29 21:41:55.660149 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:55.660116 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:55.660329 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:55.660257 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:56.660609 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:56.660350 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:56.661029 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:56.660391 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:56.661029 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:56.660711 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:56.661029 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:56.660753 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:56.844835 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:56.844800 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="05284e93accec6b8abfbb410d507b78720f39506d51cd2cc998494142664a60b" exitCode=0 Oct 29 21:41:56.845044 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:56.844855 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"05284e93accec6b8abfbb410d507b78720f39506d51cd2cc998494142664a60b"} Oct 29 21:41:57.660544 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:57.660036 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:57.660544 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:57.660191 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:58.659676 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:58.659651 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:58.659809 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:58.659781 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:41:58.659871 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:58.659820 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:58.659974 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:58.659952 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:41:58.851060 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:58.851032 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="f2c43e839152d4489c93cb084face98ea0dec11a04dbd28f6a5a3a497f3b78f0" exitCode=0 Oct 29 21:41:58.851365 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:58.851071 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"f2c43e839152d4489c93cb084face98ea0dec11a04dbd28f6a5a3a497f3b78f0"} Oct 29 21:41:59.353674 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.353455 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:41:59.353674 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.353649 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:59.353831 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.353676 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:41:59.353831 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.353680 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:59.353831 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.353762 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret podName:96fce558-209a-4a47-a807-af68663c9981 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.353729595 +0000 UTC m=+34.323942411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret") pod "global-pull-secret-syncer-z6ll7" (UID: "96fce558-209a-4a47-a807-af68663c9981") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:59.354214 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354198 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:59.354246 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354214 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:59.354246 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354230 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:59.354246 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354241 2587 projected.go:196] Error preparing data for projected volume kube-api-access-crb7h for pod openshift-network-diagnostics/network-check-target-snzg7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:59.354340 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354253 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.354240058 +0000 UTC m=+34.324453052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:59.354340 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.354273 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h podName:7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.354263277 +0000 UTC m=+34.324476109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-crb7h" (UniqueName: "kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h") pod "network-check-target-snzg7" (UID: "7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:59.660012 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.659941 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:41:59.660139 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:41:59.660047 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:41:59.856424 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.856393 2587 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="16eaedb8b77de1e3bbf83960abd7c3bfa0af9b9aee10e443170468adca005f3e" exitCode=0 Oct 29 21:41:59.857038 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.856466 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerDied","Data":"16eaedb8b77de1e3bbf83960abd7c3bfa0af9b9aee10e443170468adca005f3e"} Oct 29 21:41:59.859736 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859716 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"e7c4db22792b98f84a8c2ec34793f0e1707511467dcac6883c89f29ba2362f50"} Oct 29 21:41:59.859819 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859772 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"8ad5f3bb8c7f35d69ff9cbe3cb551ff6f5bb2ee3373febf150161bac99675be8"} Oct 29 21:41:59.859819 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859804 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"51744ea144178e4a51a8f9ec21d04f5bdfbd2ff3586e9a2872fedaee0afb02f7"} Oct 29 21:41:59.859894 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859834 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"01a7eb529de21cd911d32fadc04c25a0f4a306d7b4dcd9c1689c0ba3fbb6858f"} Oct 29 21:41:59.859894 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859869 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"5767625438c2706ebbb49045c804ebd24165aab85e2e150f68544225fa5a808f"} Oct 29 21:41:59.859971 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:41:59.859894 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"d843bf91d7fbb167021cfb85e500d9390d3387eaeb87752db7d8d523e113304e"} Oct 29 21:42:00.660408 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:00.660378 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:00.660547 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:00.660379 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:00.660547 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:00.660474 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:00.660631 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:00.660543 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:00.864509 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:00.864477 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" event={"ID":"7d8f95dd-9d27-4626-891b-e5ef541d7562","Type":"ContainerStarted","Data":"86ae568934aa55bb98264ddbc5ab8b68f2fad1b9e108a893a36c736fa3478a95"} Oct 29 21:42:00.938174 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:00.938076 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fgvh6" podStartSLOduration=8.123230324 podStartE2EDuration="18.938062748s" totalImagesPullingTime="10.814832424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:00.938042125 +0000 UTC m=+19.908254942" watchObservedRunningTime="2026-10-29 21:42:00.938062748 +0000 UTC m=+19.908275543" Oct 29 21:42:01.660659 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:01.660574 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:01.660789 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:01.660668 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:01.869666 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:01.869632 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"e59bea96706f3f0c131fee38299fe013b83862c96a6024fc250b246f8f56d5dd"} Oct 29 21:42:02.660250 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:02.660219 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:02.660426 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:02.660220 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:02.660426 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:02.660314 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:02.660426 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:02.660383 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:03.662759 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:03.662663 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:03.663207 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:03.662767 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:03.876775 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:03.876740 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" event={"ID":"bcb85288-ba12-4aa5-aa3a-33059623a339","Type":"ContainerStarted","Data":"6a932a0c020ae6bde7b550d3a895903ce7217d3b718c474f00b8f4a2127ff0ab"} Oct 29 21:42:03.877078 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:03.877053 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:03.892432 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:03.892403 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:04.034051 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.033997 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" podStartSLOduration=6.896569977 podStartE2EDuration="22.033982462s" totalImagesPullingTime="15.137412485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:04.033971219 +0000 UTC m=+23.004184036" watchObservedRunningTime="2026-10-29 21:42:04.033982462 +0000 UTC m=+23.004195259" Oct 29 21:42:04.665296 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.665255 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:04.665933 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.665798 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:04.666980 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:04.666551 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:04.666980 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:04.666033 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:04.879461 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.879433 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:04.879618 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.879517 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:04.893886 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:04.893859 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:05.533419 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.533343 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z6ll7"] Oct 29 21:42:05.533599 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.533454 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:05.533599 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:05.533553 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:05.536588 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.536563 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-snzg7"] Oct 29 21:42:05.536723 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.536652 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:05.536759 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:05.536730 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:05.537086 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.537068 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z8gwh"] Oct 29 21:42:05.537166 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:05.537156 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:05.537241 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:05.537226 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:06.660584 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:06.660546 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:06.661142 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:06.660682 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:07.495271 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:07.495240 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:07.495402 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:07.495367 2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:42:07.495996 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:07.495976 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:07.660507 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:07.660474 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:07.660673 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:07.660600 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:07.660673 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:07.660650 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:07.661161 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:07.660735 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:08.660040 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:08.660003 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:08.660299 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:08.660114 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:09.660064 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:09.660032 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:09.660064 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:09.660059 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:09.660588 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:09.660137 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:09.660588 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:09.660251 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:10.660033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:10.659837 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:10.660192 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:10.660125 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z8gwh" podUID="ecdf6506-6fca-4e05-b998-69d4d9d63316" Oct 29 21:42:11.661299 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.661268 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:11.662007 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:11.661977 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snzg7" podUID="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" Oct 29 21:42:11.662615 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.662592 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:11.662873 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:11.662851 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6ll7" podUID="96fce558-209a-4a47-a807-af68663c9981" Oct 29 21:42:11.791100 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.791068 2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeReady" Oct 29 21:42:11.791251 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.791196 2587 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 21:42:11.859257 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.859222 2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zzg6v"] Oct 29 21:42:11.862457 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.862418 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:11.863178 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.863155 2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rhgtx"] Oct 29 21:42:11.865840 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.865824 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.881984 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.881962 2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qhzlq"] Oct 29 21:42:11.885041 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.885022 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzg6v"] Oct 29 21:42:11.885127 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.885111 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:11.890895 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.890877 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:11.890895 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.890885 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:11.891033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.890982 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:11.891033 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.891014 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:11.891139 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.891040 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:11.891139 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.891065 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:11.891139 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.891126 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:11.891293 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.891273 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:11.910910 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.910883 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhgtx"] Oct 29 21:42:11.924830 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.924807 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhzlq"] Oct 29 21:42:11.938092 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938060 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:11.938092 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938094 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:11.938244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938118 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.938244 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938134 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:11.938315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938271 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.938315 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938309 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:11.938410 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938388 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vhh\" (UniqueName: \"kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.938462 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938426 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rvd\" (UniqueName: \"kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:11.938498 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938464 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.938529 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938496 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:11.938529 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.938516 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqn9s\" (UniqueName: \"kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:11.991000 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.990964 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:11.991000 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.990994 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:11.991326 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.991240 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:11.991326 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:11.991264 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:12.039494 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039466 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vhh\" (UniqueName: \"kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.039642 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039523 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rvd\" (UniqueName: \"kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:12.039642 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039558 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.039642 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039588 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.039642 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039620 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqn9s\" (UniqueName: \"kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.039837 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.039690 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:12.039837 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039778 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:12.039837 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.039789 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.539768017 +0000 UTC m=+31.509980836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : configmap "kube-rbac-proxy" not found Oct 29 21:42:12.039837 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039818 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039855 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039881 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.039955 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039967 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.039973 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.039993 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls podName:fcb75346-5362-4c07-9564-ba9344456020 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.539981016 +0000 UTC m=+31.510193827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls") pod "dns-default-qhzlq" (UID: "fcb75346-5362-4c07-9564-ba9344456020") : secret "dns-default-metrics-tls" not found Oct 29 21:42:12.040071 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.040046 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.040331 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.040093 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.540076971 +0000 UTC m=+31.510289796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.040331 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.040118 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.040331 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.040157 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.040331 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.040243 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.040557 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.040539 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.043834 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.043772 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:12.068146 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.068118 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqn9s\" (UniqueName: \"kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.068276 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.068260 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rvd\" (UniqueName: \"kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:12.068319 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.068258 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vhh\" (UniqueName: \"kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.175800 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.175757 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:12.490116 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.490084 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzg6v"] Oct 29 21:42:12.493305 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:42:12.493269 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d7b9d4_101c_4515_897c_86fdafeb2a6b.slice/crio-71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028 WatchSource:0}: Error finding container 71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028: Status 404 returned error can't find the container with id 71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028 Oct 29 21:42:12.543558 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.543519 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.543708 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.543565 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:12.543708 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.543598 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:12.543708 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543651 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:12.543840 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543732 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:13.543714602 +0000 UTC m=+32.513927415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : configmap "kube-rbac-proxy" not found Oct 29 21:42:12.543840 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543738 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:12.543840 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543738 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.543840 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543786 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls podName:fcb75346-5362-4c07-9564-ba9344456020 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:13.543774861 +0000 UTC m=+32.513987662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls") pod "dns-default-qhzlq" (UID: "fcb75346-5362-4c07-9564-ba9344456020") : secret "dns-default-metrics-tls" not found Oct 29 21:42:12.544002 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:12.543850 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:13.543832117 +0000 UTC m=+32.514044927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.659962 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.659926 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:12.676472 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.676446 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:12.676821 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.676800 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:12.897820 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:12.897739 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzg6v" event={"ID":"87d7b9d4-101c-4515-897c-86fdafeb2a6b","Type":"ContainerStarted","Data":"71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028"} Oct 29 21:42:13.551949 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.551890 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:13.552144 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.551960 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:13.552144 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.551987 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:13.552144 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552020 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:13.552144 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552089 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:13.552144 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552092 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:13.552399 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552100 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.552083393 +0000 UTC m=+34.522296191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : configmap "kube-rbac-proxy" not found Oct 29 21:42:13.552399 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552183 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls podName:fcb75346-5362-4c07-9564-ba9344456020 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.552171817 +0000 UTC m=+34.522384613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls") pod "dns-default-qhzlq" (UID: "fcb75346-5362-4c07-9564-ba9344456020") : secret "dns-default-metrics-tls" not found Oct 29 21:42:13.552399 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:13.552199 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:15.552189735 +0000 UTC m=+34.522402544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:13.663419 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.663137 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:13.663579 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.663479 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:13.680312 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.680286 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:13.692884 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.692856 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:13.692884 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.692873 2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:13.693094 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.692964 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:13.693094 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:13.693075 2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:15.146043 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.145999 2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjxpk_a3b0e867-b455-45f3-94cf-3aeae43cb6ee/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:15.369493 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.369455 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:15.369493 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.369495 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:15.369764 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.369516 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:15.369764 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.369641 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:15.369764 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.369723 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs podName:ecdf6506-6fca-4e05-b998-69d4d9d63316 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:47.369702665 +0000 UTC m=+66.339915476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs") pod "network-metrics-daemon-z8gwh" (UID: "ecdf6506-6fca-4e05-b998-69d4d9d63316") : secret "metrics-daemon-secret" not found Oct 29 21:42:15.373597 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.373569 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:15.373850 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.373629 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:15.520692 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.520651 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:15.539839 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.539812 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:15.571003 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.570968 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:15.571133 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.571022 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:15.571133 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.571093 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:15.571285 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.571249 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:15.571433 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.571321 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.571304337 +0000 UTC m=+38.541517137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:15.571433 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.571346 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:15.571433 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:15.571419 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls podName:fcb75346-5362-4c07-9564-ba9344456020 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.571401714 +0000 UTC m=+38.541614524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls") pod "dns-default-qhzlq" (UID: "fcb75346-5362-4c07-9564-ba9344456020") : secret "dns-default-metrics-tls" not found Oct 29 21:42:15.571608 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.571566 2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:15.856295 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.856223 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-snzg7"] Oct 29 21:42:15.859637 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:42:15.859607 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7500a4e8_f02f_45a4_8b8a_b34ea52f2aaf.slice/crio-851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e WatchSource:0}: Error finding container 851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e: Status 404 returned error can't find the container with id 851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e Oct 29 21:42:15.891665 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.891630 2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z6ll7"] Oct 29 21:42:15.894806 ip-10-0-141-100 kubenswrapper[2587]: W1029 21:42:15.894776 2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fce558_209a_4a47_a807_af68663c9981.slice/crio-2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad WatchSource:0}: Error finding container 2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad: Status 404 returned error can't find the container with id 2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad Oct 29 21:42:15.905196 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.905127 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-snzg7" event={"ID":"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf","Type":"ContainerStarted","Data":"851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e"} Oct 29 21:42:15.906574 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.906542 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzg6v" event={"ID":"87d7b9d4-101c-4515-897c-86fdafeb2a6b","Type":"ContainerStarted","Data":"43f9a7b734a090acc1a4b67ac6b7a433ce4e87c41634eb37688984df48684566"} Oct 29 21:42:15.907610 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.907587 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z6ll7" event={"ID":"96fce558-209a-4a47-a807-af68663c9981","Type":"ContainerStarted","Data":"2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad"} Oct 29 21:42:15.948813 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:15.948755 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zzg6v" podStartSLOduration=2.238171639 podStartE2EDuration="4.948739296s" totalImagesPullingTime="2.710567657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:15.948680699 +0000 UTC m=+34.918893517" watchObservedRunningTime="2026-10-29 21:42:15.948739296 +0000 UTC m=+34.918952108" Oct 29 21:42:18.123301 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:18.123267 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2cmw5_b57d5263-1744-42ca-ae20-2c52ab13ba81/dns-node-resolver/0.log" Oct 29 21:42:19.123012 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.122985 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gstvd_295393c7-fd50-401c-a65b-95293cc4953a/node-ca/0.log" Oct 29 21:42:19.603600 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.603567 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:19.604113 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.603613 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:19.604113 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:19.603750 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:19.604113 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:19.603763 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.604113 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:19.603841 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls podName:fcb75346-5362-4c07-9564-ba9344456020 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:27.603818933 +0000 UTC m=+46.574031743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls") pod "dns-default-qhzlq" (UID: "fcb75346-5362-4c07-9564-ba9344456020") : secret "dns-default-metrics-tls" not found Oct 29 21:42:19.604113 ip-10-0-141-100 kubenswrapper[2587]: E1029 21:42:19.603884 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:27.603868526 +0000 UTC m=+46.574081335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.920838 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.920801 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-snzg7" event={"ID":"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf","Type":"ContainerStarted","Data":"187b7073050a935147348ba44b532cd43f503bb58cc031e8ef74d38dffb2cf7b"} Oct 29 21:42:19.921052 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.920964 2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:19.963934 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:19.963863 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-snzg7" podStartSLOduration=34.534208829 podStartE2EDuration="37.963849576s" totalImagesPullingTime="3.429640747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:19.963290116 +0000 UTC m=+38.933502935" watchObservedRunningTime="2026-10-29 21:42:19.963849576 +0000 UTC m=+38.934062393" Oct 29 21:42:20.323448 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:20.323406 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzg6v_87d7b9d4-101c-4515-897c-86fdafeb2a6b/serve-healthcheck-canary/0.log" Oct 29 21:42:20.924963 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:20.924897 2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z6ll7" event={"ID":"96fce558-209a-4a47-a807-af68663c9981","Type":"ContainerStarted","Data":"5d67fc9efb853d9a6fd7bfacf73e9964cd4c9a3740e658c9d4d3dea8e39698d9"} Oct 29 21:42:20.965938 ip-10-0-141-100 kubenswrapper[2587]: I1029 21:42:20.965865 2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z6ll7" podStartSLOduration=34.108802481 podStartE2EDuration="38.965849808s" totalImagesPullingTime="4.857047327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:20.96581261 +0000 UTC m=+39.936025449" watchObservedRunningTime="2026-10-29 21:42:20.965849808 +0000 UTC m=+39.936062654" Oct 29 21:42:21.070741 ip-10-0-141-100 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 21:42:21.085375 ip-10-0-141-100 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 21:42:21.085547 ip-10-0-141-100 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 21:42:21.085605 ip-10-0-141-100 systemd[1]: kubelet.service: Consumed 2.299s CPU time. Oct 29 21:42:21.098612 ip-10-0-141-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:42:21.190634 ip-10-0-141-100 kubenswrapper[4376]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:21.190634 ip-10-0-141-100 kubenswrapper[4376]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:42:21.190634 ip-10-0-141-100 kubenswrapper[4376]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:21.190634 ip-10-0-141-100 kubenswrapper[4376]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:21.192816 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192713 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.192816 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192815 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.192816 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192819 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.192816 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192822 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192825 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192835 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192839 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192842 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192844 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192847 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192851 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192854 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192857 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192859 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192862 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192865 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192867 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192871 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192874 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192877 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192879 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192884 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.192962 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192890 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192894 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192897 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192913 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192917 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192920 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192923 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192926 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192929 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192932 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192936 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192942 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192946 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192950 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192953 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192955 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192959 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192962 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192965 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.193471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192968 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192971 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192974 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192976 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192979 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192982 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192984 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192987 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192990 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192993 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192995 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.192998 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193004 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193007 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193010 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193012 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193015 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193018 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193021 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193024 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.194206 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193026 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193029 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193031 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193034 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193036 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193040 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193043 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193045 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193048 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193051 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193054 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193057 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193060 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193062 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193065 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193068 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193070 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193073 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193075 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.194782 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193078 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193080 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193083 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193085 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193088 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193093 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193095 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193098 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193101 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193103 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193106 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193109 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193111 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193114 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193116 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193119 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193122 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193125 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193127 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193130 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.195276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193133 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193135 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193138 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193140 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193143 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.193147 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195231 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195249 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195253 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195256 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195260 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195266 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195269 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195273 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195275 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195278 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195281 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195284 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195287 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.195753 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195290 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195293 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195296 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195299 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195301 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195304 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195306 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195309 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195311 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195314 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195317 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195319 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195323 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195326 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195330 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195332 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195336 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195338 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195341 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.196250 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195344 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195348 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195351 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195354 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195356 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195359 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195362 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195364 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195367 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195369 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195372 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195375 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195378 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195380 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195383 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195385 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195388 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195391 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195402 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.196707 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195405 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195407 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195410 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195412 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195415 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195418 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195421 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195424 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195427 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195430 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195433 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195435 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195438 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195441 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195444 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195447 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195449 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195452 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195454 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195457 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.197182 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195460 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195462 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195465 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195467 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195470 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195472 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195475 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195477 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195480 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195483 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195485 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195488 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195496 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195499 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195502 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195505 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195509 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195511 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195514 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195517 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.197660 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195520 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195523 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195525 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195528 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195531 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195534 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195536 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195539 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195543 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195545 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195548 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195550 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195553 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195555 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.195558 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196123 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196131 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196134 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196136 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196140 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.198193 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196144 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196147 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196149 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196152 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196157 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196160 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196163 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196166 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196169 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196171 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196174 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196177 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196180 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196183 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196185 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196188 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196190 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196193 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196196 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196198 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.198722 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196201 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196204 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196207 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196209 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196212 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196215 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196217 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196220 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196222 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196225 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196228 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196232 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196234 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196241 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196244 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196246 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196249 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196252 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196263 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.199238 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196266 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196268 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196271 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196274 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196277 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196280 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196282 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196285 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196287 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196290 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196293 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196295 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196298 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196300 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196303 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196306 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196309 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196311 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196314 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.199700 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196317 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196320 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196322 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196325 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196328 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196331 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196333 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196338 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196341 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196343 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196346 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196349 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196352 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196355 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196357 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196360 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196364 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196368 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196371 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196374 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.200192 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196377 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196379 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196382 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196385 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196387 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196390 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196394 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196398 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196401 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196403 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196406 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196409 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196412 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196415 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196417 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196420 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196422 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196425 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196428 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.200714 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196432 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196435 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196438 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.196441 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196542 4376 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196555 4376 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196563 4376 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196568 4376 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196573 4376 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196577 4376 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196582 4376 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196587 4376 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196590 4376 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196594 4376 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196597 4376 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196601 4376 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196604 4376 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196608 4376 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196611 4376 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196614 4376 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196617 4376 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196620 4376 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196624 4376 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196627 4376 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:42:21.201190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196631 4376 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196634 4376 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196637 4376 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196641 4376 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196644 4376 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196648 4376 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196651 4376 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196654 4376 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196661 4376 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196664 4376 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196667 4376 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196671 4376 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196675 4376 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196678 4376 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196682 4376 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196686 4376 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196689 4376 flags.go:64] FLAG: --enable-server="true" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196693 4376 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196699 4376 flags.go:64] FLAG: --event-burst="100" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196702 4376 flags.go:64] FLAG: --event-qps="50" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196706 4376 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196710 4376 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196713 4376 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196723 4376 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:42:21.201771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196726 4376 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196729 4376 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196733 4376 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196736 4376 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196739 4376 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196743 4376 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196746 4376 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196749 4376 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196752 4376 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196755 4376 flags.go:64] FLAG: --feature-gates="" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196760 4376 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196763 4376 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196766 4376 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196770 4376 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196773 4376 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196776 4376 flags.go:64] FLAG: --help="false" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196781 4376 flags.go:64] FLAG: --hostname-override="ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196785 4376 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196788 4376 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196792 4376 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196795 4376 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196799 4376 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196802 4376 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:42:21.202378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196805 4376 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196809 4376 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196813 4376 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196816 4376 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196819 4376 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196822 4376 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196825 4376 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196828 4376 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196831 4376 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196834 4376 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196837 4376 flags.go:64] FLAG: --lock-file="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196840 4376 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196843 4376 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196846 4376 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196852 4376 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196855 4376 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196859 4376 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196862 4376 flags.go:64] FLAG: --logging-format="text" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196865 4376 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196868 4376 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196871 4376 flags.go:64] FLAG: --manifest-url="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196874 4376 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196879 4376 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196883 4376 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196887 4376 flags.go:64] FLAG: --max-pods="110" Oct 29 21:42:21.202948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196891 4376 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196895 4376 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196898 4376 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196918 4376 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196921 4376 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196925 4376 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196928 4376 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196941 4376 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196944 4376 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196948 4376 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196952 4376 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196955 4376 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196958 4376 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196961 4376 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196964 4376 flags.go:64] FLAG: --port="10250" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196967 4376 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196970 4376 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-083fd340b6ba3b1eb" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196974 4376 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196977 4376 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196980 4376 flags.go:64] FLAG: --register-node="true" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196983 4376 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196988 4376 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196991 4376 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196994 4376 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.196997 4376 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197001 4376 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:42:21.203600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197005 4376 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197008 4376 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197011 4376 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197014 4376 flags.go:64] FLAG: --runonce="false" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197017 4376 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197021 4376 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197026 4376 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197029 4376 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197032 4376 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197035 4376 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197039 4376 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197042 4376 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197045 4376 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197049 4376 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197052 4376 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197054 4376 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197058 4376 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197062 4376 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197065 4376 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197071 4376 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197073 4376 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197076 4376 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197081 4376 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197084 4376 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197088 4376 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:42:21.204260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197090 4376 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197094 4376 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197098 4376 flags.go:64] FLAG: --v="2" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197104 4376 flags.go:64] FLAG: --version="false" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197108 4376 flags.go:64] FLAG: --vmodule="" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197113 4376 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197116 4376 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197265 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197270 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197274 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197276 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197279 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197282 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197286 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197289 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197292 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197294 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197297 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197300 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197302 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197305 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.205096 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197307 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197310 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197313 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197323 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197326 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197329 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197332 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197337 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197340 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197344 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197347 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197349 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197352 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197355 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197358 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197362 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197366 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197370 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197374 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.205631 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197377 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197380 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197383 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197386 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197388 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197393 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197396 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197399 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197402 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197404 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197407 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197410 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197412 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197416 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197418 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197421 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197424 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197433 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197436 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197438 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.206136 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197441 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197444 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197446 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197449 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197452 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197454 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197457 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197460 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197463 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197465 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197468 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197471 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197473 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197476 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197479 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197481 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197484 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197488 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197491 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.206838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197494 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197496 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197499 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197502 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197504 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197507 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197509 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197512 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197515 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197517 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197520 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197523 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197526 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197528 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197531 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197536 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197538 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197541 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197543 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197547 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.207462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197549 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197552 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197554 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197557 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197559 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197562 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197564 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197567 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197570 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197573 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197577 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197579 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197582 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197585 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.197617 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197667 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.208212 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197672 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197675 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197679 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197681 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197684 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197687 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197690 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197693 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197697 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197699 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197702 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197706 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197709 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197712 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197715 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197717 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197720 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197723 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197726 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197729 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.208728 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197731 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197734 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197737 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197739 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197742 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197745 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197747 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197751 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197754 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197757 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197759 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197762 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197764 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197767 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197770 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197772 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197775 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197778 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197780 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197783 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.209252 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197785 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197789 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197791 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197795 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197798 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197800 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197803 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197806 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197808 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197811 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197813 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197816 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197819 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197821 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197824 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197827 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197829 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197832 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197835 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197837 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.209748 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197840 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197843 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197845 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197848 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197850 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197853 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197856 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197858 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197861 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197864 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197866 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197869 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197871 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197874 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197878 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197882 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197885 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197888 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197890 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.210405 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197893 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197895 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197898 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197913 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197917 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197920 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197924 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197926 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197929 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197932 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197934 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197937 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197943 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197947 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197950 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197953 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197956 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197958 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197961 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.211197 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197963 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197966 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197969 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197971 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197974 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197976 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.197979 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.201895 4376 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.201931 4376 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.201952 4376 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.201957 4376 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202021 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202026 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202030 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202033 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202035 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202038 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202041 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202044 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202047 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.211920 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202050 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202053 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202056 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202059 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202062 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202065 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202067 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202070 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202073 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202075 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202078 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202081 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202083 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202086 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202089 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202091 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202094 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202097 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202099 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.212488 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202102 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202105 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202110 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202113 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202115 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202118 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202121 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202124 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202127 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202129 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202133 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202137 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202141 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202144 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202147 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202149 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202152 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202155 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202157 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202160 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.213021 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202163 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202166 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202168 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202171 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202174 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202176 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202179 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202181 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202184 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202187 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202189 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202192 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202194 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202197 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202200 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202203 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202206 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202211 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202216 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.213506 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202219 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202222 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202225 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202228 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202231 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202234 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202236 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202239 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202242 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202244 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202247 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202249 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202252 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202255 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202258 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202261 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202264 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202266 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202269 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202272 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.213982 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202274 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202277 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202280 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202283 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202286 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202297 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202300 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202303 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202306 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202310 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202313 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202316 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202319 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202321 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202324 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202327 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202329 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202332 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.214474 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202335 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.202366 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202540 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202545 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202549 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202551 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202555 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202558 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202561 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202564 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202567 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202569 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202572 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202574 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202577 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202580 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.214981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202583 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202585 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202588 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202590 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202593 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202595 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202598 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202601 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202604 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202607 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202609 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202612 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202615 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202617 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202619 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202622 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202625 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202627 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202630 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202632 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.215361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202635 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202638 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202640 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202643 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202646 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202649 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202651 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202654 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202657 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202659 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202662 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202665 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202668 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202671 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202673 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202676 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202678 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202681 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202683 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.215843 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202686 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202689 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202694 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202698 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202700 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202704 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202707 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202709 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202712 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202715 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202717 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202720 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202722 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202725 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202727 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202731 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202733 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202736 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202739 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.216309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202742 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202745 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202748 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202750 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202753 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202755 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202758 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202760 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202763 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202765 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202768 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202771 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202773 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202776 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202778 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202781 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202785 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202788 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202790 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202793 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.216780 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202795 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202798 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202801 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202804 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202806 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202809 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202813 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202816 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202820 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202823 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202826 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202829 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202831 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202834 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.202857 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202893 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:21.217457 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202897 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202914 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202918 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202921 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202923 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202926 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202929 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202932 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202935 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202937 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202940 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202943 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202945 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202949 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202952 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202955 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202957 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202960 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202963 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:21.217834 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202965 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202968 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202974 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202976 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202979 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202982 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202984 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202987 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202990 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202992 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202995 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.202998 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203000 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203003 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203006 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203008 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203011 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203014 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203018 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:21.218319 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203021 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203024 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203026 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203029 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203031 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203034 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203037 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203039 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203042 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203045 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203048 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203051 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203053 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203056 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203058 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203062 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203065 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203068 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203072 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:21.218783 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203076 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203080 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203083 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203086 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203089 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203091 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203094 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203097 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203099 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203102 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203105 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203107 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203111 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203114 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203117 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203120 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203122 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203125 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203127 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203130 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:21.219387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203133 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203135 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203139 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203141 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203144 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203146 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203149 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203151 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203155 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203158 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203161 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203163 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203166 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203168 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203171 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203174 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203176 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203179 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203181 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203184 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:21.220052 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203187 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203189 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203192 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203197 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203202 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203205 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203208 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:21.203211 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.203271 4376 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204348 4376 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204408 4376 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204588 4376 server.go:1014] "Starting client certificate rotation" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204731 4376 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204790 4376 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:42 +0000 UTC" deadline="2028-07-16 19:49:37.309530646 +0000 UTC" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.204820 4376 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15022h7m16.104712418s" Oct 29 21:42:21.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.205065 4376 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:21.221000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.206189 4376 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:21.221000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.208044 4376 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:42:21.221000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.209677 4376 log.go:25] "Validated CRI v1 image API" Oct 29 21:42:21.221000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.210499 4376 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:42:21.221000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.213957 4376 fs.go:135] Filesystem UUIDs: map[26e1c196-46c3-457d-b7f0-fe6fde7ebb88:/dev/nvme0n1p3 6937126b-5e4a-4f6e-b0b3-cf619608a308:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 29 21:42:21.221229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.213988 4376 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11cb51a1-45ef-4b15-b6f7-74cf138af237/volumes/kubernetes.io~projected/kube-api-access-2n896:{mountpoint:/var/lib/kubelet/pods/11cb51a1-45ef-4b15-b6f7-74cf138af237/volumes/kubernetes.io~projected/kube-api-access-2n896 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/295393c7-fd50-401c-a65b-95293cc4953a/volumes/kubernetes.io~projected/kube-api-access-jq9fp:{mountpoint:/var/lib/kubelet/pods/295393c7-fd50-401c-a65b-95293cc4953a/volumes/kubernetes.io~projected/kube-api-access-jq9fp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29e456b9-c486-4c7b-8339-a87159a8fe7e/volumes/kubernetes.io~projected/kube-api-access-ntjs5:{mountpoint:/var/lib/kubelet/pods/29e456b9-c486-4c7b-8339-a87159a8fe7e/volumes/kubernetes.io~projected/kube-api-access-ntjs5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf/volumes/kubernetes.io~projected/kube-api-access-crb7h:{mountpoint:/var/lib/kubelet/pods/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf/volumes/kubernetes.io~projected/kube-api-access-crb7h major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d8f95dd-9d27-4626-891b-e5ef541d7562/volumes/kubernetes.io~projected/kube-api-access-wzw7g:{mountpoint:/var/lib/kubelet/pods/7d8f95dd-9d27-4626-891b-e5ef541d7562/volumes/kubernetes.io~projected/kube-api-access-wzw7g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~projected/kube-api-access-x7rvd:{mountpoint:/var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~projected/kube-api-access-x7rvd major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96fce558-209a-4a47-a807-af68663c9981/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/96fce558-209a-4a47-a807-af68663c9981/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~projected/kube-api-access-79mrb:{mountpoint:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~projected/kube-api-access-79mrb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3b0e867-b455-45f3-94cf-3aeae43cb6ee/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a3b0e867-b455-45f3-94cf-3aeae43cb6ee/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57d5263-1744-42ca-ae20-2c52ab13ba81/volumes/kubernetes.io~projected/kube-api-access-zfktv:{mountpoint:/var/lib/kubelet/pods/b57d5263-1744-42ca-ae20-2c52ab13ba81/volumes/kubernetes.io~projected/kube-api-access-zfktv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~projected/kube-api-access-gccm7:{mountpoint:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~projected/kube-api-access-gccm7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be983072-fbb9-4919-8cba-be3765773830/volumes/kubernetes.io~projected/kube-api-access-n7vhh:{mountpoint:/var/lib/kubelet/pods/be983072-fbb9-4919-8cba-be3765773830/volumes/kubernetes.io~projected/kube-api-access-n7vhh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecdf6506-6fca-4e05-b998-69d4d9d63316/volumes/kubernetes.io~projected/kube-api-access-8zhbl:{mountpoint:/var/lib/kubelet/pods/ecdf6506-6fca-4e05-b998-69d4d9d63316/volumes/kubernetes.io~projected/kube-api-access-8zhbl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbf7bd76-73c6-4cf0-bad9-f7830153f335/volumes/kubernetes.io~projected/kube-api-access-482h2:{mountpoint:/var/lib/kubelet/pods/fbf7bd76-73c6-4cf0-bad9-f7830153f335/volumes/kubernetes.io~projected/kube-api-access-482h2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb75346-5362-4c07-9564-ba9344456020/volumes/kubernetes.io~projected/kube-api-access-wqn9s:{mountpoint:/var/lib/kubelet/pods/fcb75346-5362-4c07-9564-ba9344456020/volumes/kubernetes.io~projected/kube-api-access-wqn9s major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ba76ae570d9fed50a97d0885018a406cf8c667b13112abd90093a2986e957a52/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b6256c1a7749e05ed517fec46b2fef3517a121015b09514c1adda86793fedc68/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f2d5928e95da5d6e3b10f7c83057b89a8e3b4143ca521cfdcfa1b11c20005c6e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/de1c0a9894d3d3a160abdb5ce9414081ffad6ebcf1e40160338ca6904f27fd09/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a60d460795681b8881908dd0c8bff524a34b848f7412b1020ebad3933503ee89/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a7b5623a1369e16f8227ae7e50fd47d00cdc5469ce90efd534b2528ef2425409/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/565069e211e1ee79a5d7e055b5d8b8e25058f0846f95bba556f7d6d2f72ce304/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/45d1be3aedeec346be3ed34e9e3e37c8414b6c6b165cb9d674c3b5b8e424c1d6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/56145eae6983fa68bf65a2e4950a59a00a9ac94ce984142d5dcb75031b050fd6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/806cb203849e94284b6417716d3b55bb5ee5b1b5f748b31b7cb701a6cfabec80/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1f2766245a5183e6b82ce3be784df7f1d8916664d4aef54000ccde541efbf76d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/7d69f3db77a11b1b4d6cab6753f0b6b17256d8bd370b6303ff1b853792690b1a/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d1e33b715c9282d05bb3408d7ca27e40a3a837a7c75e1749a43488978a0a2248/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f292cde3f22890e8bca6a9f9d7747b91ae701a2a47c9b28936a6e5b908e420d1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/218fbf4c7aee51e37b4f45c9eb2cf775675245554e476e56dda030df3805b002/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b823dcd9116e24f87510f9e699896f21f2e11b53b43f634f16f29ded9106c176/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/608200dc54573bea441e8dd50074b4eb77557aa70d0291d4ffeece95da7c5274/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c4d4b24104365ac454c7a910704a2a0b81426a19a94d6420006d813b5028efbb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a01fd4d463144316d1c1c11914c8ff0a8bd5a936a445e020881af5d30f8ace6e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/bd0f1fded71674b8521d1a2fe8c3cb506883d1e549abcf649df62d489fef2c1e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bbbb7ada82bb4f9075b2bb837958e5cdefc64800c0234c11e229d369c7c4400b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/8d640e0b38c29a61fafeef87e7f9a6b7ba1446edb7e51c3b7099c768bf2b10e3/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/671c673687f437cfc4fbcadb0f40bf046a3d8ef8ff6ace40c4ec48b848bbd154/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2047bde9fc8d2de40294aaf8ec20a50f3a330aa895e8e8cd0a478ad4fc46b6a8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/a745a66222807d4d6d9137af348e9048fb2e75d4775ce9d44a4a597171e0eabf/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/0a9bd08077dabef31277c4332d7787a2875c55fb8c4b6c842cc95a9cbebabd5d/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c1f00dc8c0d3aac9ebdc772deb94a975062b3a5dcd0c425b2e1a650f36b02744/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/4b9e3152094bd5c14e92ab490dc4279a4520bc45e761360fd56eb9039a8f9859/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2628d9d8e3c8fabd0ffb7f7e00545801c0ee8959c873d5a0f2d1fdb7713e4a89/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c94d586012ec8a7e5767d2f72fcef3e496f2613117797f8c10a98589c9e567a7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8b87b749873f8329c061dea11ad0fd33d233428612018274c3abe9313545bf64/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/cd139fb7caddbf09f07dbac213f54220a731b259cc4c08bdb245a292abfa3f0b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5406a1fbe904cff12d78b01ce4335944e50671d06db0e53971c3b90808747b64/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/65045e3d83b85f1fd5e603e994fcdbfe39e158dbb7266dffbc73f54c135d3de8/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/19dccaea3cb8cacf930807b851e80ea00a41d77154bcb7691c7718a33d8defbf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4659bdd33104eb3eeea02bdff9adaf15d9011355b29e705dadca4a8c9d069400/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/30eadced2cfd5779041633a555e41b9c7b7ebfa4dbebc1972931c4e161745585/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 21:42:21.221229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.220470 4376 manager.go:217] Machine: {Timestamp:2026-10-29 21:42:21.219987043 +0000 UTC m=+0.068463899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124453 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f1ac009482ea940e546ef9ff9381 SystemUUID:ec29f1ac-0094-82ea-940e-546ef9ff9381 BootID:9070598f-f861-448a-9134-9b6317d2a467 Filesystems:[{Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf/volumes/kubernetes.io~projected/kube-api-access-crb7h DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8e55e0c93fe830f8534d99469cee174acfdc9f687f00f601bc2bf7074b66c04/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29e456b9-c486-4c7b-8339-a87159a8fe7e/volumes/kubernetes.io~projected/kube-api-access-ntjs5 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volumes/kubernetes.io~projected/kube-api-access-gccm7 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1fd0038dc3779575f61af0028e7997a998058bd72fc8cf0f6239f30ebe077a8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~projected/kube-api-access-x7rvd DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57d5263-1744-42ca-ae20-2c52ab13ba81/volumes/kubernetes.io~projected/kube-api-access-zfktv DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ad8b511289deef7078d977654551b9520b1c38b9a89498572e7f17cf87fa74/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87d7b9d4-101c-4515-897c-86fdafeb2a6b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~projected/kube-api-access-79mrb DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/264c4cebe58d14d83ffbded8eaf617f431466d970c76fc5c2ff1b719705e620c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcb85288-ba12-4aa5-aa3a-33059623a339/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d8f95dd-9d27-4626-891b-e5ef541d7562/volumes/kubernetes.io~projected/kube-api-access-wzw7g DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11cb51a1-45ef-4b15-b6f7-74cf138af237/volumes/kubernetes.io~projected/kube-api-access-2n896 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecdf6506-6fca-4e05-b998-69d4d9d63316/volumes/kubernetes.io~projected/kube-api-access-8zhbl DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3b0e867-b455-45f3-94cf-3aeae43cb6ee/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2af106f57ead52f1d38d2875e8c9a4e93a9d090b72d595d7f5bbcac56dd9af4d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be983072-fbb9-4919-8cba-be3765773830/volumes/kubernetes.io~projected/kube-api-access-n7vhh DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbf7bd76-73c6-4cf0-bad9-f7830153f335/volumes/kubernetes.io~projected/kube-api-access-482h2 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/295393c7-fd50-401c-a65b-95293cc4953a/volumes/kubernetes.io~projected/kube-api-access-jq9fp DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96fce558-209a-4a47-a807-af68663c9981/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/851ffa425ab7fa181771fec58bce0645740d18e65e0ceefda4caacd45e3add3e/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb75346-5362-4c07-9564-ba9344456020/volumes/kubernetes.io~projected/kube-api-access-wqn9s DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71887d54b60b15e2d1877fce9b2fa387815a79453c4645de1fead2ddaac26028/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a69a97cb792b0d29392127861b1683bf4a148e625295f8f4ae17a01fe608b309/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f07771b63e3e93d9036c895d5815484593eed62ec772df84f0f134ca51744ad/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9885f82f-5fef-44bd-84b4-de771bc1e0e7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43179c3e42d5127d50f33d94b8f5193dc0540ab400cb58e36a67e2fdaa34051d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7021b233e1ddd08862f5bda6d43af82abee9d84e9dd6af842cc445c24b5202b6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2f07771b63e3e93 MacAddress:7e:fe:43:97:e8:70 Speed:10000 Mtu:8901} {Name:71887d54b60b15e MacAddress:16:dc:46:90:f1:ec Speed:10000 Mtu:8901} {Name:851ffa425ab7fa1 MacAddress:de:5b:1f:26:94:78 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c6:6c:42:b3 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:59:d7:3d:d4:d8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c6:6c:42:b3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:5c:fc:68:5b:a1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:07:3f:90:0c:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:42:21.221229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.220725 4376 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.220775 4376 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221107 4376 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221137 4376 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221281 4376 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221289 4376 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221305 4376 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221327 4376 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:21.221498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221442 4376 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:42:21.221723 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221628 4376 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:42:21.221750 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221743 4376 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:42:21.221794 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221782 4376 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:42:21.221823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221798 4376 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:42:21.221823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221810 4376 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:42:21.221823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221820 4376 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:42:21.221948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.221829 4376 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:42:21.222651 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.222627 4376 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:21.222651 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.222646 4376 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:21.223322 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.223304 4376 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:42:21.226023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226003 4376 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:21.226066 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226023 4376 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:21.226177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226165 4376 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 21:42:21.226448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226433 4376 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:42:21.226484 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226468 4376 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:42:21.226595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226585 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:42:21.226624 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226600 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:42:21.226624 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226606 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:42:21.226624 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226614 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:42:21.226624 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226623 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226632 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226639 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226645 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226652 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226658 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226673 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226683 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226722 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:42:21.226739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.226732 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:42:21.230347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.230330 4376 server.go:1263] "Started kubelet" Oct 29 21:42:21.230447 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.230400 4376 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:42:21.230960 ip-10-0-141-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:42:21.234193 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.234167 4376 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:42:21.235949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.235853 4376 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:42:21.236078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.235985 4376 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:42:21.236138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236085 4376 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:42:21.236138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236097 4376 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:42:21.236234 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236135 4376 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:42:21.236326 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.236307 4376 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-100.ec2.internal\" not found" Oct 29 21:42:21.236433 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236403 4376 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:42:21.237302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236517 4376 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:42:21.237302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236603 4376 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:44 +0000 UTC" deadline="2028-06-06 21:17:31.048757364 +0000 UTC" Oct 29 21:42:21.237302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236645 4376 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14063h35m9.812116331s" Oct 29 21:42:21.237302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.236743 4376 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:42:21.238826 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238785 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d8f95dd-9d27-4626-891b-e5ef541d7562" volumeName="kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238831 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d8f95dd-9d27-4626-891b-e5ef541d7562" volumeName="kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238847 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9885f82f-5fef-44bd-84b4-de771bc1e0e7" volumeName="kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238864 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb85288-ba12-4aa5-aa3a-33059623a339" volumeName="kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238877 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be983072-fbb9-4919-8cba-be3765773830" volumeName="kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238889 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbf7bd76-73c6-4cf0-bad9-f7830153f335" volumeName="kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238922 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7b9d4-101c-4515-897c-86fdafeb2a6b" volumeName="kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238936 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96fce558-209a-4a47-a807-af68663c9981" volumeName="kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret" seLinuxMountContext="" Oct 29 21:42:21.238948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238948 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9885f82f-5fef-44bd-84b4-de771bc1e0e7" volumeName="kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238963 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb85288-ba12-4aa5-aa3a-33059623a339" volumeName="kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238975 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb85288-ba12-4aa5-aa3a-33059623a339" volumeName="kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238988 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb85288-ba12-4aa5-aa3a-33059623a339" volumeName="kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.238999 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb85288-ba12-4aa5-aa3a-33059623a339" volumeName="kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239011 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d8f95dd-9d27-4626-891b-e5ef541d7562" volumeName="kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239026 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be983072-fbb9-4919-8cba-be3765773830" volumeName="kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239034 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29e456b9-c486-4c7b-8339-a87159a8fe7e" volumeName="kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239042 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57d5263-1744-42ca-ae20-2c52ab13ba81" volumeName="kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239049 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb75346-5362-4c07-9564-ba9344456020" volumeName="kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239058 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7b9d4-101c-4515-897c-86fdafeb2a6b" volumeName="kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239072 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9885f82f-5fef-44bd-84b4-de771bc1e0e7" volumeName="kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239085 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecdf6506-6fca-4e05-b998-69d4d9d63316" volumeName="kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239091 4376 factory.go:55] Registering systemd factory Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239115 4376 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239099 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb75346-5362-4c07-9564-ba9344456020" volumeName="kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239180 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11cb51a1-45ef-4b15-b6f7-74cf138af237" volumeName="kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239222 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b0e867-b455-45f3-94cf-3aeae43cb6ee" volumeName="kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239239 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11cb51a1-45ef-4b15-b6f7-74cf138af237" volumeName="kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239253 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf" volumeName="kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239267 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d8f95dd-9d27-4626-891b-e5ef541d7562" volumeName="kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239280 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be983072-fbb9-4919-8cba-be3765773830" volumeName="kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239306 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb75346-5362-4c07-9564-ba9344456020" volumeName="kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239319 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29e456b9-c486-4c7b-8339-a87159a8fe7e" volumeName="kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239331 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29e456b9-c486-4c7b-8339-a87159a8fe7e" volumeName="kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5" seLinuxMountContext="" Oct 29 21:42:21.239327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239343 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b0e867-b455-45f3-94cf-3aeae43cb6ee" volumeName="kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca" seLinuxMountContext="" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239356 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57d5263-1744-42ca-ae20-2c52ab13ba81" volumeName="kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv" seLinuxMountContext="" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239369 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="295393c7-fd50-401c-a65b-95293cc4953a" volumeName="kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca" seLinuxMountContext="" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239403 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="295393c7-fd50-401c-a65b-95293cc4953a" volumeName="kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp" seLinuxMountContext="" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239415 4376 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239420 4376 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239599 4376 factory.go:153] Registering CRI-O factory Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239617 4376 factory.go:223] Registration of the crio container factory successfully Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239691 4376 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239716 4376 factory.go:103] Registering Raw factory Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.239732 4376 manager.go:1199] Started watching for new ooms in manager Oct 29 21:42:21.240445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.240032 4376 manager.go:319] Starting recovery of all containers Oct 29 21:42:21.262268 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.262222 4376 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:42:21.263780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.263754 4376 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:42:21.263780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.263784 4376 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:42:21.263996 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.263806 4376 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:42:21.263996 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.263846 4376 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 21:42:21.293499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293451 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="16eaedb8b77de1e3bbf83960abd7c3bfa0af9b9aee10e443170468adca005f3e" exitCode=0 Oct 29 21:42:21.293499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293493 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="f2c43e839152d4489c93cb084face98ea0dec11a04dbd28f6a5a3a497f3b78f0" exitCode=0 Oct 29 21:42:21.293499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293506 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="05284e93accec6b8abfbb410d507b78720f39506d51cd2cc998494142664a60b" exitCode=0 Oct 29 21:42:21.293769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293520 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="87a4c1bc944f5957b8ec8ec0d038df2b06692555030d3526f304fa26aef02717" exitCode=0 Oct 29 21:42:21.293769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293530 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="e732f391bdead3fda4af7d5b2e14cde677ac1c0f94f985832f7b298690cd95df" exitCode=0 Oct 29 21:42:21.293769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.293540 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d8f95dd-9d27-4626-891b-e5ef541d7562" containerID="c0473c44e6e5b0edf59daaabc19ed3e3361a5f982954e885ffd86335b59d44f4" exitCode=0 Oct 29 21:42:21.294047 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.294028 4376 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:21.294047 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.294038 4376 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 21:42:21.297729 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.297687 4376 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:21.298657 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.298625 4376 generic.go:356] "Generic (PLEG): container finished" podID="97accf019645b7962e49aedeb58f438f" containerID="ef6e0238c91f1a72a1cf81ff98597419bd76bfcf3e257d7a5dcd04174b43a1ba" exitCode=0 Oct 29 21:42:21.306663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.306634 4376 manager.go:324] Recovery completed Oct 29 21:42:21.306791 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.306743 4376 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:21.311272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.311249 4376 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:21.321721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321662 4376 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:42:21.321721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321678 4376 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:42:21.321721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321697 4376 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321824 4376 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321834 4376 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321852 4376 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321857 4376 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321862 4376 policy_none.go:50] "Start" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321870 4376 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:42:21.321881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.321879 4376 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:42:21.322089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.322014 4376 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 21:42:21.322089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.322021 4376 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 21:42:21.322089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.322026 4376 policy_none.go:44] "Start" Oct 29 21:42:21.326166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.326133 4376 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:42:21.329442 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.329424 4376 manager.go:341] "Starting Device Plugin manager" Oct 29 21:42:21.329950 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.329458 4376 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:42:21.330046 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.329954 4376 server.go:87] "Starting device plugin registration server" Oct 29 21:42:21.330272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330258 4376 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:42:21.330352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330273 4376 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:42:21.330405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330380 4376 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:42:21.330521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330485 4376 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:42:21.330521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330524 4376 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:42:21.330521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330531 4376 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:42:21.330730 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.330594 4376 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:42:21.330512448Z","UUID":"96349ced-bd2d-4d1b-9b75-b71037cd9040","Handler":null,"Name":"","Endpoint":""} Oct 29 21:42:21.331722 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.331698 4376 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:42:21.332120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.332105 4376 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:42:21.332210 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.332161 4376 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:42:21.364453 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.364401 4376 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal"] Oct 29 21:42:21.364888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.364858 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" event={"ID":"4c9c0102f0b78e22ae4ea8593b59d168","Type":"ContainerStarted","Data":"eca6484930ff55305de3a36d4f69327d74e7346f4b8fb42981121cfc13ac2f8c"} Oct 29 21:42:21.364973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.364896 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" event={"ID":"4c9c0102f0b78e22ae4ea8593b59d168","Type":"ContainerStarted","Data":"ba08f9d41806d25d02b748162fdd483750b21a31eef530da12d788ed450c1339"} Oct 29 21:42:21.365033 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.364985 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerStarted","Data":"f85d3b5f5b0051cab9d66ac0ff18e9904d22f45565538fd3a0b1b85fb30b4457"} Oct 29 21:42:21.365033 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.364997 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.365107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.365101 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.365151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.365005 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerDied","Data":"ef6e0238c91f1a72a1cf81ff98597419bd76bfcf3e257d7a5dcd04174b43a1ba"} Oct 29 21:42:21.365193 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.365159 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" event={"ID":"97accf019645b7962e49aedeb58f438f","Type":"ContainerStarted","Data":"c80f160034404ddf1eed4d4445b7dfc0ebc2b3cb70964dc40470df98ba321352"} Oct 29 21:42:21.391337 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.391308 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:21.391438 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.391366 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.392914 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.392888 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:21.392968 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:21.392950 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.430402 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.430360 4376 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:42:21.431763 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.431742 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:42:21.431877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.431777 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:42:21.431877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.431793 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:42:21.431877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.431832 4376 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.475797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.475757 4376 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.475993 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.475852 4376 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.495566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.495525 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.495566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.495569 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.495934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.495593 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596185 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596252 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596275 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596527 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596527 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596326 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c9c0102f0b78e22ae4ea8593b59d168-config\") pod \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" (UID: \"4c9c0102f0b78e22ae4ea8593b59d168\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:21.596527 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:21.596377 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97accf019645b7962e49aedeb58f438f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" (UID: \"97accf019645b7962e49aedeb58f438f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:22.222783 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.222730 4376 apiserver.go:52] "Watching apiserver" Oct 29 21:42:22.268004 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.267967 4376 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:42:22.268442 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.268414 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z6ll7","openshift-cluster-node-tuning-operator/tuned-qlm4k","openshift-ingress-canary/ingress-canary-zzg6v","openshift-dns/node-resolver-2cmw5","openshift-ovn-kubernetes/ovnkube-node-zs2sf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl","openshift-multus/network-metrics-daemon-z8gwh","openshift-dns/dns-default-qhzlq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal","kube-system/konnectivity-agent-qjxpk","kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal","openshift-image-registry/node-ca-gstvd","openshift-insights/insights-runtime-extractor-rhgtx","openshift-network-diagnostics/network-check-target-snzg7","openshift-multus/multus-xbs4b","openshift-multus/multus-additional-cni-plugins-fgvh6","openshift-network-operator/iptables-alerter-7xwz2"] Oct 29 21:42:22.273655 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.273625 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.275585 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.275559 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.275974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.275959 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.285053 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285031 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.285155 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285031 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.285424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285410 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.285678 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285662 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.285746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285720 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.285895 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285882 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:42:22.285952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.285921 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.286363 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.286348 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.286426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.286413 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.286865 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.286844 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:42:22.286865 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.286868 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.287061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.286937 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:42:22.287145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287129 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.287532 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287516 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:42:22.287643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287628 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:42:22.287681 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287646 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:22.287718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287699 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:42:22.287718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287700 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:42:22.287718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287714 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:42:22.287797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287712 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:42:22.287797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.287759 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:42:22.300029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300009 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.300296 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300275 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300297 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300117 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300140 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300183 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300345 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300232 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300047 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300151 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300322 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300308 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300394 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:22.300403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300211 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:42:22.300866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300766 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.300866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300806 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:22.300866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300807 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:42:22.300866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300844 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:42:22.300866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300847 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:22.301052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300872 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:22.301052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300935 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.301052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.300941 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:22.301052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301030 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.301052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301043 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:42:22.301192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301080 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:42:22.301192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301124 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:42:22.301354 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301334 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:22.301354 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301346 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:22.301517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301364 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:22.301517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301396 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.301517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301396 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:22.301517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301457 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:22.301641 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301542 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:22.301873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301857 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:22.301950 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301898 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:22.301950 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.301886 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:42:22.327871 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.327842 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:22.328011 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:22.327893 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-100.ec2.internal" Oct 29 21:42:22.329324 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.329310 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:22.329386 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:22.329343 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-100.ec2.internal" Oct 29 21:42:22.337735 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.337716 4376 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:42:22.399369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399327 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.399369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399370 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.399586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399395 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzw7g\" (UniqueName: \"kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.399586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.399586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399514 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-tuned\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.399756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399656 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqn9s\" (UniqueName: \"kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.399807 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.399793 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb75346-5362-4c07-9564-ba9344456020-config-volume\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.400174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400153 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.400301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400185 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vhh\" (UniqueName: \"kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.400301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400208 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.400301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400225 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.400301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400245 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.400301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400288 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400312 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400328 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400325 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be983072-fbb9-4919-8cba-be3765773830-data-volume\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400374 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400467 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.400517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400498 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfktv\" (UniqueName: \"kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.400785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.400785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400618 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.400785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400718 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.400785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.400982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400861 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-env-overrides\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.400982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400862 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.400982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.400982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.400979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401003 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401027 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401051 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401072 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401099 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n896\" (UniqueName: \"kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-daemon-config\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401223 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.401272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401256 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401275 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401335 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9fp\" (UniqueName: \"kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401364 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401386 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401437 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401470 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.401573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401494 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401605 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401637 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401675 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-agent-certs\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401694 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57d5263-1744-42ca-ae20-2c52ab13ba81-tmp-dir\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401678 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401722 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401755 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401844 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.401869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401872 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401919 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401943 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401970 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.401976 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/295393c7-fd50-401c-a65b-95293cc4953a-serviceca\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhbl\" (UniqueName: \"kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402052 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb75346-5362-4c07-9564-ba9344456020-tmp-dir\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402071 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402105 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d8f95dd-9d27-4626-891b-e5ef541d7562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402108 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402151 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402168 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402188 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402201 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402217 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402276 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402316 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402345 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402385 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482h2\" (UniqueName: \"kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402493 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29e456b9-c486-4c7b-8339-a87159a8fe7e-cni-binary-copy\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.402551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402522 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402581 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402599 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccm7\" (UniqueName: \"kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402624 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402662 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402679 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96fce558-209a-4a47-a807-af68663c9981-original-pull-secret\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402718 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rvd\" (UniqueName: \"kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402765 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9885f82f-5fef-44bd-84b4-de771bc1e0e7-tmp\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402781 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-config\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.402873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402821 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402944 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402987 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bcb85288-ba12-4aa5-aa3a-33059623a339-ovn-node-metrics-cert\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.402989 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403054 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403073 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be983072-fbb9-4919-8cba-be3765773830-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403094 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403131 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403194 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403232 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403273 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403303 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjs5\" (UniqueName: \"kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.403347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403323 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.403836 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403408 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.403836 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.403783 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11cb51a1-45ef-4b15-b6f7-74cf138af237-iptables-alerter-script\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.404466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404442 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mrb\" (UniqueName: \"kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.404537 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404483 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:22.404585 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404561 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.404631 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404596 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.404683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404628 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.404920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bcb85288-ba12-4aa5-aa3a-33059623a339-ovnkube-script-lib\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.404977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.404958 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.405025 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405000 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:22.405069 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405030 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.405069 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405062 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.405157 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405087 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.405157 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405109 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.405425 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405407 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0e867-b455-45f3-94cf-3aeae43cb6ee-konnectivity-ca\") pod \"konnectivity-agent-qjxpk\" (UID: \"a3b0e867-b455-45f3-94cf-3aeae43cb6ee\") " pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:22.405865 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.405846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87d7b9d4-101c-4515-897c-86fdafeb2a6b-cert\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:22.424578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.424535 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vhh\" (UniqueName: \"kubernetes.io/projected/be983072-fbb9-4919-8cba-be3765773830-kube-api-access-n7vhh\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.424714 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.424657 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqn9s\" (UniqueName: \"kubernetes.io/projected/fcb75346-5362-4c07-9564-ba9344456020-kube-api-access-wqn9s\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.424810 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.424791 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzw7g\" (UniqueName: \"kubernetes.io/projected/7d8f95dd-9d27-4626-891b-e5ef541d7562-kube-api-access-wzw7g\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.429031 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.429005 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n896\" (UniqueName: \"kubernetes.io/projected/11cb51a1-45ef-4b15-b6f7-74cf138af237-kube-api-access-2n896\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.429159 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.429116 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfktv\" (UniqueName: \"kubernetes.io/projected/b57d5263-1744-42ca-ae20-2c52ab13ba81-kube-api-access-zfktv\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.429159 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.429140 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9fp\" (UniqueName: \"kubernetes.io/projected/295393c7-fd50-401c-a65b-95293cc4953a-kube-api-access-jq9fp\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.430314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.430273 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjs5\" (UniqueName: \"kubernetes.io/projected/29e456b9-c486-4c7b-8339-a87159a8fe7e-kube-api-access-ntjs5\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.434370 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.434347 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhbl\" (UniqueName: \"kubernetes.io/projected/ecdf6506-6fca-4e05-b998-69d4d9d63316-kube-api-access-8zhbl\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.434590 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.434571 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crb7h\" (UniqueName: \"kubernetes.io/projected/7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf-kube-api-access-crb7h\") pod \"network-check-target-snzg7\" (UID: \"7500a4e8-f02f-45a4-8b8a-b34ea52f2aaf\") " pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:22.434752 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.434731 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccm7\" (UniqueName: \"kubernetes.io/projected/bcb85288-ba12-4aa5-aa3a-33059623a339-kube-api-access-gccm7\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.435037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.435020 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482h2\" (UniqueName: \"kubernetes.io/projected/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kube-api-access-482h2\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.435081 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.435067 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rvd\" (UniqueName: \"kubernetes.io/projected/87d7b9d4-101c-4515-897c-86fdafeb2a6b-kube-api-access-x7rvd\") pod \"ingress-canary-zzg6v\" (UID: \"87d7b9d4-101c-4515-897c-86fdafeb2a6b\") " pod="openshift-ingress-canary/ingress-canary-zzg6v" Oct 29 21:42:22.439603 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.439584 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mrb\" (UniqueName: \"kubernetes.io/projected/9885f82f-5fef-44bd-84b4-de771bc1e0e7-kube-api-access-79mrb\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.505828 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505732 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.505828 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505773 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.505828 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505798 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505870 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-netd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505870 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-var-lib-kubelet\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505923 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b57d5263-1744-42ca-ae20-2c52ab13ba81-hosts-file\") pod \"node-resolver-2cmw5\" (UID: \"b57d5263-1744-42ca-ae20-2c52ab13ba81\") " pod="openshift-dns/node-resolver-2cmw5" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.505977 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506029 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-k8s-cni-cncf-io\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506135 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506174 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506201 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506225 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-dbus\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506258 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506268 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-system-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506283 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506252 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-netns\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506275 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-multus-certs\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-kubernetes\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506330 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506354 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506374 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-node-log\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506374 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506409 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506433 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11cb51a1-45ef-4b15-b6f7-74cf138af237-host-slash\") pod \"iptables-alerter-7xwz2\" (UID: \"11cb51a1-45ef-4b15-b6f7-74cf138af237\") " pod="openshift-network-operator/iptables-alerter-7xwz2" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506438 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506445 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-systemd\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506410 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-host\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506463 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-systemd-units\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506463 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506445 4376 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506498 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-kubelet\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506516 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506502 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-slash\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506457 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-conf\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506540 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506567 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-system-cni-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-systemd\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506584 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506603 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506620 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506657 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506676 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506678 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-run-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506679 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506691 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506713 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-multus\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506711 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-cni-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506718 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-run-netns\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506716 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506751 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506755 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-etc-kubernetes\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506765 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.506754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506753 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-run\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506788 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506798 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-hostroot\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506804 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506818 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506822 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506836 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96fce558-209a-4a47-a807-af68663c9981-kubelet-config\") pod \"global-pull-secret-syncer-z6ll7\" (UID: \"96fce558-209a-4a47-a807-af68663c9981\") " pod="kube-system/global-pull-secret-syncer-z6ll7" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506850 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-log-socket\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506855 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506860 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-sys\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506882 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506923 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-device-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506923 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506946 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506965 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-cnibin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.506981 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507004 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507020 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysconfig\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507028 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507053 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507066 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be983072-fbb9-4919-8cba-be3765773830-crio-socket\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507075 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-host-var-lib-cni-bin\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507083 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507057 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-modprobe-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507128 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-socket-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507132 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-sys-fs\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507141 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-etc-selinux\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507173 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbf7bd76-73c6-4cf0-bad9-f7830153f335-registration-dir\") pod \"aws-ebs-csi-driver-node-m79zl\" (UID: \"fbf7bd76-73c6-4cf0-bad9-f7830153f335\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m79zl" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507173 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507205 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-ovn\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507227 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507246 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507271 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-conf-dir\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507307 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507344 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507371 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-etc-sysctl-d\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507380 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/295393c7-fd50-401c-a65b-95293cc4953a-host\") pod \"node-ca-gstvd\" (UID: \"295393c7-fd50-401c-a65b-95293cc4953a\") " pod="openshift-image-registry/node-ca-gstvd" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507374 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507410 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-run-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507413 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507423 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-var-lib-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507440 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507467 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507492 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507518 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9885f82f-5fef-44bd-84b4-de771bc1e0e7-lib-modules\") pod \"tuned-qlm4k\" (UID: \"9885f82f-5fef-44bd-84b4-de771bc1e0e7\") " pod="openshift-cluster-node-tuning-operator/tuned-qlm4k" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507519 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507550 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-cni-bin\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507560 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29e456b9-c486-4c7b-8339-a87159a8fe7e-multus-socket-dir-parent\") pod \"multus-xbs4b\" (UID: \"29e456b9-c486-4c7b-8339-a87159a8fe7e\") " pod="openshift-multus/multus-xbs4b" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507564 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:22.507570 4376 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507597 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d8f95dd-9d27-4626-891b-e5ef541d7562-cnibin\") pod \"multus-additional-cni-plugins-fgvh6\" (UID: \"7d8f95dd-9d27-4626-891b-e5ef541d7562\") " pod="openshift-multus/multus-additional-cni-plugins-fgvh6" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507621 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507643 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-host-kubelet\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.507593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bcb85288-ba12-4aa5-aa3a-33059623a339-etc-openvswitch\") pod \"ovnkube-node-zs2sf\" (UID: \"bcb85288-ba12-4aa5-aa3a-33059623a339\") " pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:22.508126 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:22.507671 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.007653481 +0000 UTC m=+1.856130321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:22.509567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.508797 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcb75346-5362-4c07-9564-ba9344456020-metrics-tls\") pod \"dns-default-qhzlq\" (UID: \"fcb75346-5362-4c07-9564-ba9344456020\") " pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.509567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.509184 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecdf6506-6fca-4e05-b998-69d4d9d63316-metrics-certs\") pod \"network-metrics-daemon-z8gwh\" (UID: \"ecdf6506-6fca-4e05-b998-69d4d9d63316\") " pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.574386 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.574351 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z8gwh" Oct 29 21:42:22.576187 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.576161 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:22.924161 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.924079 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z8gwh"] Oct 29 21:42:22.927553 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:22.927511 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdf6506_6fca_4e05_b998_69d4d9d63316.slice/crio-b01679bdabaf403699ddb7ad89a9876feb945d02eaa2efdc9de5dbc5ac0ad46e WatchSource:0}: Error finding container b01679bdabaf403699ddb7ad89a9876feb945d02eaa2efdc9de5dbc5ac0ad46e: Status 404 returned error can't find the container with id b01679bdabaf403699ddb7ad89a9876feb945d02eaa2efdc9de5dbc5ac0ad46e Oct 29 21:42:22.929933 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.929892 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:42:22.957842 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:22.957809 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhzlq"] Oct 29 21:42:22.961530 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:22.961507 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb75346_5362_4c07_9564_ba9344456020.slice/crio-856130197fce234865b23fd2113b59c0054e82105146a3ffbd74bd9616942382 WatchSource:0}: Error finding container 856130197fce234865b23fd2113b59c0054e82105146a3ffbd74bd9616942382: Status 404 returned error can't find the container with id 856130197fce234865b23fd2113b59c0054e82105146a3ffbd74bd9616942382 Oct 29 21:42:23.012001 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:23.011958 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:23.012158 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:23.012115 4376 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:23.012193 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:23.012183 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls podName:be983072-fbb9-4919-8cba-be3765773830 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:24.012168288 +0000 UTC m=+2.860645125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhgtx" (UID: "be983072-fbb9-4919-8cba-be3765773830") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:23.255547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:23.255516 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjxpk" Oct 29 21:42:23.305267 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:23.305233 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z8gwh" event={"ID":"ecdf6506-6fca-4e05-b998-69d4d9d63316","Type":"ContainerStarted","Data":"b01679bdabaf403699ddb7ad89a9876feb945d02eaa2efdc9de5dbc5ac0ad46e"} Oct 29 21:42:23.306749 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:23.306706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhzlq" event={"ID":"fcb75346-5362-4c07-9564-ba9344456020","Type":"ContainerStarted","Data":"856130197fce234865b23fd2113b59c0054e82105146a3ffbd74bd9616942382"} Oct 29 21:42:24.020035 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:24.019990 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:24.022943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:24.022890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be983072-fbb9-4919-8cba-be3765773830-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhgtx\" (UID: \"be983072-fbb9-4919-8cba-be3765773830\") " pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:24.077029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:24.076985 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhgtx" Oct 29 21:42:24.405960 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:24.405798 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhgtx"] Oct 29 21:42:25.317965 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:25.317920 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhzlq" event={"ID":"fcb75346-5362-4c07-9564-ba9344456020","Type":"ContainerStarted","Data":"7b875277a9a4a5e2d0f481c8b2d847268d4bb7977924f02b74755ff1c1d03f7a"} Oct 29 21:42:25.319696 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:25.319662 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhgtx" event={"ID":"be983072-fbb9-4919-8cba-be3765773830","Type":"ContainerStarted","Data":"4c36b5e7eb6adcc19f5a0c398491b700545b6f4732eb08dcefc29dbd15c70c84"} Oct 29 21:42:25.319838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:25.319720 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhgtx" event={"ID":"be983072-fbb9-4919-8cba-be3765773830","Type":"ContainerStarted","Data":"c58bcaf338fdeefb19f04d7fa7dc4366522f0d4d4ae1b4030b4431c193e10d1a"} Oct 29 21:42:25.321995 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:25.321968 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z8gwh" event={"ID":"ecdf6506-6fca-4e05-b998-69d4d9d63316","Type":"ContainerStarted","Data":"dce5f00d529c4ec94cefd2ac4560e5792889a302b48aa3ffe036e9dbe40916c6"} Oct 29 21:42:25.322107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:25.322014 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z8gwh" event={"ID":"ecdf6506-6fca-4e05-b998-69d4d9d63316","Type":"ContainerStarted","Data":"cd304b0b438f8c12d9925a1d07c7a58ec7c71b8ea7c4ddc19e4ef41b6c01b684"} Oct 29 21:42:26.255308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:26.255275 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:26.329251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:26.329190 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhzlq" event={"ID":"fcb75346-5362-4c07-9564-ba9344456020","Type":"ContainerStarted","Data":"7105aa4e9eff145d46d3dc82ab3eac3b3ee947b2e3fc7a92c0b9bdd5ef453fba"} Oct 29 21:42:26.329517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:26.329319 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:26.332020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:26.331979 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhgtx" event={"ID":"be983072-fbb9-4919-8cba-be3765773830","Type":"ContainerStarted","Data":"612b26318e406cfcfc65bc2bd1d15940bee7f5fcc09a1a47b8faeacdb07e44a0"} Oct 29 21:42:27.337348 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:27.337314 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhgtx" event={"ID":"be983072-fbb9-4919-8cba-be3765773830","Type":"ContainerStarted","Data":"8c5de552e64426838e5473d75ec7b8f99ea88cb3679fbcad465b492d78824808"} Oct 29 21:42:30.181631 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:30.181598 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:30.182050 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:30.181792 4376 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:42:32.029508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:32.029477 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:33.210446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.210414 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr"] Oct 29 21:42:33.212463 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.212436 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:33.223226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.223201 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sh62q"] Oct 29 21:42:33.226486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.226454 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.233562 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.233368 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 29 21:42:33.233702 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.233683 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kwgs6\"" Oct 29 21:42:33.237495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.237468 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr"] Oct 29 21:42:33.246846 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.246814 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 21:42:33.247006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.246873 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 21:42:33.252696 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.252666 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sh62q"] Oct 29 21:42:33.333965 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.333930 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-b99mh\"" Oct 29 21:42:33.375293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.375245 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba8c45b-b622-482b-960c-9fdcfbdde069-tmp\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.375293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.375298 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjt6\" (UniqueName: \"kubernetes.io/projected/7ba8c45b-b622-482b-960c-9fdcfbdde069-kube-api-access-gkjt6\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.375517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.375406 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fe6c154-5f39-4b6d-922a-3b63fdda4b77-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-z85qr\" (UID: \"4fe6c154-5f39-4b6d-922a-3b63fdda4b77\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:33.475948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.475842 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fe6c154-5f39-4b6d-922a-3b63fdda4b77-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-z85qr\" (UID: \"4fe6c154-5f39-4b6d-922a-3b63fdda4b77\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:33.475948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.475894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba8c45b-b622-482b-960c-9fdcfbdde069-tmp\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.475948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.475944 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjt6\" (UniqueName: \"kubernetes.io/projected/7ba8c45b-b622-482b-960c-9fdcfbdde069-kube-api-access-gkjt6\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.476355 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.476332 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba8c45b-b622-482b-960c-9fdcfbdde069-tmp\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.478545 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.478520 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4fe6c154-5f39-4b6d-922a-3b63fdda4b77-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-z85qr\" (UID: \"4fe6c154-5f39-4b6d-922a-3b63fdda4b77\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:33.501151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.501118 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjt6\" (UniqueName: \"kubernetes.io/projected/7ba8c45b-b622-482b-960c-9fdcfbdde069-kube-api-access-gkjt6\") pod \"downloads-6dd789c8b7-sh62q\" (UID: \"7ba8c45b-b622-482b-960c-9fdcfbdde069\") " pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.528417 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.528381 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:33.538396 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.538353 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:33.828367 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.828279 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr"] Oct 29 21:42:33.842471 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:33.842442 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe6c154_5f39_4b6d_922a_3b63fdda4b77.slice/crio-59671e3a575e0d3b82b3bcf4831cf271f92b75a85e37ff6ce16be6a6c7bcacf1 WatchSource:0}: Error finding container 59671e3a575e0d3b82b3bcf4831cf271f92b75a85e37ff6ce16be6a6c7bcacf1: Status 404 returned error can't find the container with id 59671e3a575e0d3b82b3bcf4831cf271f92b75a85e37ff6ce16be6a6c7bcacf1 Oct 29 21:42:33.854919 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:33.854876 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sh62q"] Oct 29 21:42:33.858099 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:33.858073 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba8c45b_b622_482b_960c_9fdcfbdde069.slice/crio-e785b8142450282309653d09955652d321444d6d888fd4cdc9d91804c1047901 WatchSource:0}: Error finding container e785b8142450282309653d09955652d321444d6d888fd4cdc9d91804c1047901: Status 404 returned error can't find the container with id e785b8142450282309653d09955652d321444d6d888fd4cdc9d91804c1047901 Oct 29 21:42:34.361104 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:34.361066 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sh62q" event={"ID":"7ba8c45b-b622-482b-960c-9fdcfbdde069","Type":"ContainerStarted","Data":"e785b8142450282309653d09955652d321444d6d888fd4cdc9d91804c1047901"} Oct 29 21:42:34.362984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:34.362951 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" event={"ID":"4fe6c154-5f39-4b6d-922a-3b63fdda4b77","Type":"ContainerStarted","Data":"59671e3a575e0d3b82b3bcf4831cf271f92b75a85e37ff6ce16be6a6c7bcacf1"} Oct 29 21:42:35.368383 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:35.368343 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" event={"ID":"4fe6c154-5f39-4b6d-922a-3b63fdda4b77","Type":"ContainerStarted","Data":"f8dabae1d301334eff078c73772bca1ed117aad1ee84f09e3a3c5c4a29c47ff0"} Oct 29 21:42:35.368951 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:35.368554 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:35.374659 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:35.374632 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" Oct 29 21:42:35.411814 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:35.411749 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-z85qr" podStartSLOduration=1.4808003840000001 podStartE2EDuration="2.411731413s" totalImagesPullingTime="930.931029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:35.411281563 +0000 UTC m=+14.259758412" watchObservedRunningTime="2026-10-29 21:42:35.411731413 +0000 UTC m=+14.260208257" Oct 29 21:42:36.340109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.340079 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qhzlq" Oct 29 21:42:36.437134 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.437103 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8"] Oct 29 21:42:36.439302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.439276 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.456479 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.456447 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 29 21:42:36.456602 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.456579 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9sfpc\"" Oct 29 21:42:36.456671 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.456650 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 21:42:36.456930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.456888 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 29 21:42:36.457009 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.456957 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 21:42:36.457065 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.457022 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 21:42:36.468796 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.468770 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8"] Oct 29 21:42:36.597632 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.597546 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.597632 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.597583 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.597830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.597635 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jhr\" (UniqueName: \"kubernetes.io/projected/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-kube-api-access-l4jhr\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.597830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.597694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.698260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.698228 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.698260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.698268 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jhr\" (UniqueName: \"kubernetes.io/projected/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-kube-api-access-l4jhr\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.698522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.698301 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.698522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.698357 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.699302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.699229 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.701196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.701175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.701306 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.701182 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.723873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.723846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jhr\" (UniqueName: \"kubernetes.io/projected/0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8-kube-api-access-l4jhr\") pod \"prometheus-operator-9cd89b74c-zhnq8\" (UID: \"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:36.752665 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:36.752639 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" Oct 29 21:42:37.057295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:37.057260 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8"] Oct 29 21:42:37.060985 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:37.060960 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8d9ea6_2ace_4d4c_8b19_2d3289dd44e8.slice/crio-f4d04fac345988dc822957612a7ff8ed262eadf9acb6d789e1dab5b3bced8a72 WatchSource:0}: Error finding container f4d04fac345988dc822957612a7ff8ed262eadf9acb6d789e1dab5b3bced8a72: Status 404 returned error can't find the container with id f4d04fac345988dc822957612a7ff8ed262eadf9acb6d789e1dab5b3bced8a72 Oct 29 21:42:37.377157 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:37.377063 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" event={"ID":"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8","Type":"ContainerStarted","Data":"f4d04fac345988dc822957612a7ff8ed262eadf9acb6d789e1dab5b3bced8a72"} Oct 29 21:42:37.682349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:37.682271 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zs2sf" Oct 29 21:42:38.393228 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:38.393190 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" event={"ID":"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8","Type":"ContainerStarted","Data":"dc0ceb439e01c3e55488ef7a9de181eba6bb20741e35d2607d47e62affe92166"} Oct 29 21:42:39.399071 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:39.399022 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" event={"ID":"0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8","Type":"ContainerStarted","Data":"7c51565097f3863667532596690c7f8cd96b5747c73716c386da292f4939c061"} Oct 29 21:42:39.444210 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:39.444156 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zhnq8" podStartSLOduration=2.219873289 podStartE2EDuration="3.444141254s" totalImagesPullingTime="1.224267965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:39.443948719 +0000 UTC m=+18.292425566" watchObservedRunningTime="2026-10-29 21:42:39.444141254 +0000 UTC m=+18.292618094" Oct 29 21:42:42.022273 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.022231 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d"] Oct 29 21:42:42.025158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.025136 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.057113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.057075 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 21:42:42.057317 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.057110 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x9xbz\"" Oct 29 21:42:42.057317 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.057163 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 21:42:42.057989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.057961 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d"] Oct 29 21:42:42.092718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.092683 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-59tnw"] Oct 29 21:42:42.095137 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.095113 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.127990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.127959 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5"] Oct 29 21:42:42.130218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.130196 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.140028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.139999 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.140156 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.140046 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcsz\" (UniqueName: \"kubernetes.io/projected/06fb2d9b-8ddf-4189-acae-554a0dda925d-kube-api-access-4mcsz\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.140156 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.140136 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.140255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.140198 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06fb2d9b-8ddf-4189-acae-554a0dda925d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.147329 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.147308 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 21:42:42.147588 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.147565 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dwjqp\"" Oct 29 21:42:42.147728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.147708 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 21:42:42.154594 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.154455 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 21:42:42.154594 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.154535 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tgsfd\"" Oct 29 21:42:42.156096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.156075 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 21:42:42.156894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.156869 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5"] Oct 29 21:42:42.191577 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.191538 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 21:42:42.191761 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.191689 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 21:42:42.241359 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241324 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.241359 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241366 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcsz\" (UniqueName: \"kubernetes.io/projected/06fb2d9b-8ddf-4189-acae-554a0dda925d-kube-api-access-4mcsz\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.241595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241394 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-metrics-client-ca\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.241595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06fb2d9b-8ddf-4189-acae-554a0dda925d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.241595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241484 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.241595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241539 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-tls\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.241595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241592 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241620 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9c6\" (UniqueName: \"kubernetes.io/projected/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-api-access-dn9c6\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241640 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-root\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241659 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241702 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-textfile\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241778 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.241843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241841 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241860 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241876 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hmh\" (UniqueName: \"kubernetes.io/projected/736d51e6-a9df-4c00-829e-cc256fdef7b6-kube-api-access-z5hmh\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241927 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-wtmp\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.241966 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-sys\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.242166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.242136 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06fb2d9b-8ddf-4189-acae-554a0dda925d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.244331 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.244301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.244490 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.244469 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06fb2d9b-8ddf-4189-acae-554a0dda925d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.268384 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.268354 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcsz\" (UniqueName: \"kubernetes.io/projected/06fb2d9b-8ddf-4189-acae-554a0dda925d-kube-api-access-4mcsz\") pod \"openshift-state-metrics-7bd7489957-rpt8d\" (UID: \"06fb2d9b-8ddf-4189-acae-554a0dda925d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.339807 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.339730 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" Oct 29 21:42:42.342996 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.342973 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343007 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hmh\" (UniqueName: \"kubernetes.io/projected/736d51e6-a9df-4c00-829e-cc256fdef7b6-kube-api-access-z5hmh\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343051 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343091 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-wtmp\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343117 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-sys\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343211 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-metrics-client-ca\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343241 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343268 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-tls\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343294 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343319 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9c6\" (UniqueName: \"kubernetes.io/projected/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-api-access-dn9c6\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343343 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-root\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343341 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-wtmp\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343371 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-textfile\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343399 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343418 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-sys\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343437 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.343614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343577 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.344049 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.343775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.344497 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.344236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.344497 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.344248 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.344497 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.344296 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/736d51e6-a9df-4c00-829e-cc256fdef7b6-root\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.344497 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.344316 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736d51e6-a9df-4c00-829e-cc256fdef7b6-metrics-client-ca\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.344761 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.344518 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-textfile\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.346890 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.346870 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.347233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.347211 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.347295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.347277 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.348566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.348544 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/736d51e6-a9df-4c00-829e-cc256fdef7b6-node-exporter-tls\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.368888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.368855 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hmh\" (UniqueName: \"kubernetes.io/projected/736d51e6-a9df-4c00-829e-cc256fdef7b6-kube-api-access-z5hmh\") pod \"node-exporter-59tnw\" (UID: \"736d51e6-a9df-4c00-829e-cc256fdef7b6\") " pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.369753 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.369702 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9c6\" (UniqueName: \"kubernetes.io/projected/b228e418-7c35-4d4d-b7b1-67fdaa60b43c-kube-api-access-dn9c6\") pod \"kube-state-metrics-5dc8d7d89b-2fgt5\" (UID: \"b228e418-7c35-4d4d-b7b1-67fdaa60b43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.410604 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.410568 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59tnw" Oct 29 21:42:42.422969 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:42.422932 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736d51e6_a9df_4c00_829e_cc256fdef7b6.slice/crio-fff3a4c7a4e06172aa01c8253840f01d9dfc8ce2d63a333b28c15650473a511d WatchSource:0}: Error finding container fff3a4c7a4e06172aa01c8253840f01d9dfc8ce2d63a333b28c15650473a511d: Status 404 returned error can't find the container with id fff3a4c7a4e06172aa01c8253840f01d9dfc8ce2d63a333b28c15650473a511d Oct 29 21:42:42.444149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.444115 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" Oct 29 21:42:42.658640 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.658558 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d"] Oct 29 21:42:42.662754 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:42.662716 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fb2d9b_8ddf_4189_acae_554a0dda925d.slice/crio-37dd63527f48aaeacc48dacbf4f4508594382e4b4599eee1637c3e1b6846be7f WatchSource:0}: Error finding container 37dd63527f48aaeacc48dacbf4f4508594382e4b4599eee1637c3e1b6846be7f: Status 404 returned error can't find the container with id 37dd63527f48aaeacc48dacbf4f4508594382e4b4599eee1637c3e1b6846be7f Oct 29 21:42:42.752525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:42.752493 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5"] Oct 29 21:42:42.761226 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:42.761193 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228e418_7c35_4d4d_b7b1_67fdaa60b43c.slice/crio-2a198d9d5745deb3c454c8a3d33dc8477e0cd72f922e733582cad4061f78d8f1 WatchSource:0}: Error finding container 2a198d9d5745deb3c454c8a3d33dc8477e0cd72f922e733582cad4061f78d8f1: Status 404 returned error can't find the container with id 2a198d9d5745deb3c454c8a3d33dc8477e0cd72f922e733582cad4061f78d8f1 Oct 29 21:42:43.416892 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.416849 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tnw" event={"ID":"736d51e6-a9df-4c00-829e-cc256fdef7b6","Type":"ContainerStarted","Data":"fff3a4c7a4e06172aa01c8253840f01d9dfc8ce2d63a333b28c15650473a511d"} Oct 29 21:42:43.418503 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.418475 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" event={"ID":"b228e418-7c35-4d4d-b7b1-67fdaa60b43c","Type":"ContainerStarted","Data":"2a198d9d5745deb3c454c8a3d33dc8477e0cd72f922e733582cad4061f78d8f1"} Oct 29 21:42:43.420734 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.420706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" event={"ID":"06fb2d9b-8ddf-4189-acae-554a0dda925d","Type":"ContainerStarted","Data":"eb1e5010eea21d6e7e024f8c264590a0aca7c46a8339f46d0364c55015629fea"} Oct 29 21:42:43.420854 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.420761 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" event={"ID":"06fb2d9b-8ddf-4189-acae-554a0dda925d","Type":"ContainerStarted","Data":"853ead6fc9238873c70c3450c4dbe5631d6255ea74342c3768180451ae4d2e6e"} Oct 29 21:42:43.420854 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.420781 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" event={"ID":"06fb2d9b-8ddf-4189-acae-554a0dda925d","Type":"ContainerStarted","Data":"37dd63527f48aaeacc48dacbf4f4508594382e4b4599eee1637c3e1b6846be7f"} Oct 29 21:42:43.445954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.443962 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:42:43.447925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.447882 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.467281 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467255 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 21:42:43.467458 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467276 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467320 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467352 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467385 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467417 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467460 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 21:42:43.467638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.467457 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-78gx5\"" Oct 29 21:42:43.471527 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.471506 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 21:42:43.492435 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.492390 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 21:42:43.494328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.494273 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:42:43.552705 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552678 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552717 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552735 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552775 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552800 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.552825 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552816 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552832 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552848 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlgk\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552880 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552918 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552935 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.553026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.552950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653592 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653554 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653608 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653634 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653658 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653714 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653736 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.653772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653766 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.654132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653797 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.654132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653845 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.654132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653875 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.654132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653897 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.654132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.653945 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlgk\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.657845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.655973 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.657845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.656265 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.657845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.656842 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.661325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.661298 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.665866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.665841 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.666956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.666880 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.667425 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.667395 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.667534 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.667448 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.667878 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.667858 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.668197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.668060 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.668929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.668871 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.673983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.673948 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.689218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.689186 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlgk\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\") pod \"alertmanager-main-0\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:43.776971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:43.776930 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:42:44.126510 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.126480 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:42:44.131696 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:44.131664 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac8c644_70b8_46cc_b12d_182853a63661.slice/crio-5d2654b335a770d453e210ea120d2c9215b2f6e87b44bffb528c6d9a6f701dac WatchSource:0}: Error finding container 5d2654b335a770d453e210ea120d2c9215b2f6e87b44bffb528c6d9a6f701dac: Status 404 returned error can't find the container with id 5d2654b335a770d453e210ea120d2c9215b2f6e87b44bffb528c6d9a6f701dac Oct 29 21:42:44.425912 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.425805 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"5d2654b335a770d453e210ea120d2c9215b2f6e87b44bffb528c6d9a6f701dac"} Oct 29 21:42:44.435339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.435304 4376 generic.go:356] "Generic (PLEG): container finished" podID="736d51e6-a9df-4c00-829e-cc256fdef7b6" containerID="dd8182d98af5f0c7bfbd1f96fff1d31bb0b13cd23624b7ae2734adec8f327d25" exitCode=0 Oct 29 21:42:44.435495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.435420 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tnw" event={"ID":"736d51e6-a9df-4c00-829e-cc256fdef7b6","Type":"ContainerDied","Data":"dd8182d98af5f0c7bfbd1f96fff1d31bb0b13cd23624b7ae2734adec8f327d25"} Oct 29 21:42:44.438088 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.438060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" event={"ID":"b228e418-7c35-4d4d-b7b1-67fdaa60b43c","Type":"ContainerStarted","Data":"c643ee2418396c9ce99477191808c21ca4ddf79b42a27bff2b156b80cbcce791"} Oct 29 21:42:44.438199 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.438114 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" event={"ID":"b228e418-7c35-4d4d-b7b1-67fdaa60b43c","Type":"ContainerStarted","Data":"c56b20fab58e2e75ec4ac2ed1c95fac68a3b100caf109077272a218c9356aa47"} Oct 29 21:42:44.438199 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.438146 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" event={"ID":"b228e418-7c35-4d4d-b7b1-67fdaa60b43c","Type":"ContainerStarted","Data":"d22f25b36d0281810e2f62897b6b5f6fd2f4f825bb39bde85ba2a9cca4357150"} Oct 29 21:42:44.440857 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.440831 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" event={"ID":"06fb2d9b-8ddf-4189-acae-554a0dda925d","Type":"ContainerStarted","Data":"c6b03eb57cff8474f93863c57f0baffb303c94bfe8533956d236353afa1e49b8"} Oct 29 21:42:44.525113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.525066 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2fgt5" podStartSLOduration=1.232666439 podStartE2EDuration="2.525051081s" totalImagesPullingTime="1.292384642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:44.524668677 +0000 UTC m=+23.373145525" watchObservedRunningTime="2026-10-29 21:42:44.525051081 +0000 UTC m=+23.373527927" Oct 29 21:42:44.963188 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.963128 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rpt8d" podStartSLOduration=2.876003007 podStartE2EDuration="3.963109068s" totalImagesPullingTime="1.087106061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:44.572390702 +0000 UTC m=+23.420867549" watchObservedRunningTime="2026-10-29 21:42:44.963109068 +0000 UTC m=+23.811585912" Oct 29 21:42:44.963414 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.963393 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:42:44.966300 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.966279 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:44.982980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.982954 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 21:42:44.982980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.982976 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 21:42:44.983173 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.983070 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zgx4c\"" Oct 29 21:42:44.983830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.983809 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 21:42:44.983971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.983836 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 21:42:44.983971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.983836 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 21:42:44.991351 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:44.991327 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:42:45.066522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066489 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066688 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066541 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066688 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066570 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066770 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066728 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066824 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066787 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066878 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066861 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.066943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.066891 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgpc\" (UniqueName: \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168095 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168135 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgpc\" (UniqueName: \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168193 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168219 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168239 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168534 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168288 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.168872 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.168851 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.170191 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.169436 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.170191 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.169891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.170191 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.170135 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.171872 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.171812 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.172109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.172092 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.195003 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.194966 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgpc\" (UniqueName: \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\") pod \"console-c994468bf-jdtdd\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.279824 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.279745 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:45.447162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.447123 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235" exitCode=0 Oct 29 21:42:45.447645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.447209 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235"} Oct 29 21:42:45.450464 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.450438 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tnw" event={"ID":"736d51e6-a9df-4c00-829e-cc256fdef7b6","Type":"ContainerStarted","Data":"11ee1e30dcbbbd51134882b23a5897ca9e8fa1fa649aa8d4ace308991bd7a67e"} Oct 29 21:42:45.450556 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.450501 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59tnw" event={"ID":"736d51e6-a9df-4c00-829e-cc256fdef7b6","Type":"ContainerStarted","Data":"818865e39bcfbd976c5995a5ccb4027b9a7cfed33f31660fc1d97cbbdef6d719"} Oct 29 21:42:45.549192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.549096 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-59tnw" podStartSLOduration=2.346692507 podStartE2EDuration="3.549081529s" totalImagesPullingTime="1.11116619s" totalInitContainerRuntime="91.222832ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:45.548966477 +0000 UTC m=+24.397443346" watchObservedRunningTime="2026-10-29 21:42:45.549081529 +0000 UTC m=+24.397558372" Oct 29 21:42:45.604077 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:45.604034 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:42:45.607891 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:45.607864 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1eb0cc_d288_431e_8888_4f12a3eeedec.slice/crio-b55dacf26f59ba306dc47ae1b56a72ee738a801a72bc6bb4407618e5679d73eb WatchSource:0}: Error finding container b55dacf26f59ba306dc47ae1b56a72ee738a801a72bc6bb4407618e5679d73eb: Status 404 returned error can't find the container with id b55dacf26f59ba306dc47ae1b56a72ee738a801a72bc6bb4407618e5679d73eb Oct 29 21:42:46.444657 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:42:46.444618 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:42:46.456775 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.456740 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c994468bf-jdtdd" event={"ID":"fc1eb0cc-d288-431e-8888-4f12a3eeedec","Type":"ContainerStarted","Data":"b55dacf26f59ba306dc47ae1b56a72ee738a801a72bc6bb4407618e5679d73eb"} Oct 29 21:42:46.530926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.525896 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cb667f678-wvsbj"] Oct 29 21:42:46.535947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.535890 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.556837 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.556805 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cb667f678-wvsbj"] Oct 29 21:42:46.567759 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.567729 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 21:42:46.567947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.567797 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 21:42:46.568031 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.567956 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 21:42:46.568094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.568081 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 21:42:46.568223 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.568200 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8k6fd\"" Oct 29 21:42:46.568374 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.568335 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-73ibbd5is65ug\"" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682585 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-metrics-server-audit-profiles\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682636 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxccb\" (UniqueName: \"kubernetes.io/projected/2f8ca511-4b4b-4448-a680-21c4cffbc095-kube-api-access-zxccb\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-client-certs\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682716 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-client-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682741 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-tls\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f8ca511-4b4b-4448-a680-21c4cffbc095-audit-log\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.683007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.682785 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784117 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-metrics-server-audit-profiles\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784166 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxccb\" (UniqueName: \"kubernetes.io/projected/2f8ca511-4b4b-4448-a680-21c4cffbc095-kube-api-access-zxccb\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784216 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-client-certs\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784241 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-client-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784267 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-tls\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f8ca511-4b4b-4448-a680-21c4cffbc095-audit-log\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.784544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.784321 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.785154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.785131 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.786167 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.786023 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2f8ca511-4b4b-4448-a680-21c4cffbc095-metrics-server-audit-profiles\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.786519 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.786494 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2f8ca511-4b4b-4448-a680-21c4cffbc095-audit-log\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.789504 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.789481 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-client-certs\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.792330 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.792285 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-client-ca-bundle\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.800453 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.800422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2f8ca511-4b4b-4448-a680-21c4cffbc095-secret-metrics-server-tls\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.814899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.814861 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxccb\" (UniqueName: \"kubernetes.io/projected/2f8ca511-4b4b-4448-a680-21c4cffbc095-kube-api-access-zxccb\") pod \"metrics-server-6cb667f678-wvsbj\" (UID: \"2f8ca511-4b4b-4448-a680-21c4cffbc095\") " pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.852583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.852544 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:42:46.858697 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.857433 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6"] Oct 29 21:42:46.862704 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.861993 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:46.882028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.881991 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6"] Oct 29 21:42:46.882189 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.882115 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 21:42:46.882189 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.882115 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p7dcd\"" Oct 29 21:42:46.985742 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.985670 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrp2\" (UniqueName: \"kubernetes.io/projected/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-kube-api-access-gcrp2\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:46.985742 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:46.985713 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.086643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.086559 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrp2\" (UniqueName: \"kubernetes.io/projected/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-kube-api-access-gcrp2\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.086643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.086609 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.089720 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.089689 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.113332 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.113291 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrp2\" (UniqueName: \"kubernetes.io/projected/a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00-kube-api-access-gcrp2\") pod \"monitoring-plugin-77f4f76594-wjjc6\" (UID: \"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.189182 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.189150 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cb667f678-wvsbj"] Oct 29 21:42:47.195372 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:47.195341 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8ca511_4b4b_4448_a680_21c4cffbc095.slice/crio-cc7c77843360c58cbb74f84e5ab29a751f809972b0ca41d0b2b4a2a16f47ddf8 WatchSource:0}: Error finding container cc7c77843360c58cbb74f84e5ab29a751f809972b0ca41d0b2b4a2a16f47ddf8: Status 404 returned error can't find the container with id cc7c77843360c58cbb74f84e5ab29a751f809972b0ca41d0b2b4a2a16f47ddf8 Oct 29 21:42:47.221267 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.221241 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:47.462733 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.462703 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" event={"ID":"2f8ca511-4b4b-4448-a680-21c4cffbc095","Type":"ContainerStarted","Data":"cc7c77843360c58cbb74f84e5ab29a751f809972b0ca41d0b2b4a2a16f47ddf8"} Oct 29 21:42:47.466816 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.466785 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65"} Oct 29 21:42:47.466943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.466868 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5"} Oct 29 21:42:47.466943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.466933 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8"} Oct 29 21:42:47.561513 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:47.561212 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6"] Oct 29 21:42:47.569832 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:47.569790 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e75a2e_b02d_4fc2_b60c_6c7d085e7a00.slice/crio-27d6bb632c4ef4cb82307b877efb1cfb8312c5a447bf120fb2736aa40bba6d11 WatchSource:0}: Error finding container 27d6bb632c4ef4cb82307b877efb1cfb8312c5a447bf120fb2736aa40bba6d11: Status 404 returned error can't find the container with id 27d6bb632c4ef4cb82307b877efb1cfb8312c5a447bf120fb2736aa40bba6d11 Oct 29 21:42:48.481769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:48.481730 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08"} Oct 29 21:42:48.482253 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:48.481842 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1"} Oct 29 21:42:48.483360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:48.483307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" event={"ID":"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00","Type":"ContainerStarted","Data":"27d6bb632c4ef4cb82307b877efb1cfb8312c5a447bf120fb2736aa40bba6d11"} Oct 29 21:42:49.491495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:49.491465 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerStarted","Data":"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4"} Oct 29 21:42:49.587572 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:49.587517 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.964174107 podStartE2EDuration="6.587502686s" totalImagesPullingTime="3.604210148s" totalInitContainerRuntime="19.118431ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:49.573876764 +0000 UTC m=+28.422353635" watchObservedRunningTime="2026-10-29 21:42:49.587502686 +0000 UTC m=+28.435979531" Oct 29 21:42:50.497427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.497389 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" event={"ID":"a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00","Type":"ContainerStarted","Data":"26849ac98a3600e5fdcaf0fb15583db25e53542d971a74d85f82d12856c2c896"} Oct 29 21:42:50.497926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.497457 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:50.499542 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.499514 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" event={"ID":"2f8ca511-4b4b-4448-a680-21c4cffbc095","Type":"ContainerStarted","Data":"3512b9c9e9f18ff5c01a3891f320a8a53f852b7900780807f69cbc0ecdcd3ae6"} Oct 29 21:42:50.501446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.501417 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c994468bf-jdtdd" event={"ID":"fc1eb0cc-d288-431e-8888-4f12a3eeedec","Type":"ContainerStarted","Data":"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3"} Oct 29 21:42:50.503679 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.503656 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" Oct 29 21:42:50.539922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.539852 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wjjc6" podStartSLOduration=2.177719882 podStartE2EDuration="4.539837904s" totalImagesPullingTime="2.362118022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:50.539766135 +0000 UTC m=+29.388242983" watchObservedRunningTime="2026-10-29 21:42:50.539837904 +0000 UTC m=+29.388314749" Oct 29 21:42:50.587526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.587471 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c994468bf-jdtdd" podStartSLOduration=2.25053727 podStartE2EDuration="6.587457006s" totalImagesPullingTime="4.336919736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:50.58643435 +0000 UTC m=+29.434911197" watchObservedRunningTime="2026-10-29 21:42:50.587457006 +0000 UTC m=+29.435933852" Oct 29 21:42:50.677341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:50.677276 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" podStartSLOduration=2.445405169 podStartE2EDuration="4.677256822s" totalImagesPullingTime="2.231851653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:50.676273221 +0000 UTC m=+29.524750067" watchObservedRunningTime="2026-10-29 21:42:50.677256822 +0000 UTC m=+29.525733667" Oct 29 21:42:52.032238 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:52.032208 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-snzg7" Oct 29 21:42:52.512028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:52.511941 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sh62q" event={"ID":"7ba8c45b-b622-482b-960c-9fdcfbdde069","Type":"ContainerStarted","Data":"a772deeb9de3f0c6cbda964c6aad37757cd905d9ff15aef9f50c5e1f694b6572"} Oct 29 21:42:52.512341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:52.512194 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:52.524538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:52.524505 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sh62q" Oct 29 21:42:52.558763 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:52.558699 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-sh62q" podStartSLOduration=1.503198695 podStartE2EDuration="19.558680828s" totalImagesPullingTime="18.055482133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:52.557471031 +0000 UTC m=+31.405947877" watchObservedRunningTime="2026-10-29 21:42:52.558680828 +0000 UTC m=+31.407157675" Oct 29 21:42:53.594287 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.594252 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:42:53.625238 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.625202 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:42:53.625423 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.625330 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.645969 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.645939 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.646132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.645985 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.646533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.646010 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.647239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.646852 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhql5\" (UniqueName: \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.648047 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.648009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.648241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.648216 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.649132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.648315 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.649132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.648354 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.681130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.681083 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 29 21:42:53.750926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.750862 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.750926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.750931 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.751148 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.750954 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.751400 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.751379 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.751754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.751725 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.751968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.751949 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.752107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.752087 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhql5\" (UniqueName: \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.752277 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.752260 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.752406 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.752390 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.753140 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.753114 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.753240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.751738 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.753774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.753750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.754210 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.752310 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.755216 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.755190 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.755713 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.755692 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.781329 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.781298 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhql5\" (UniqueName: \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\") pod \"console-6df8854f7d-rzpj7\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:53.950029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:53.949934 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:42:54.324240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:54.324192 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:42:54.328256 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:42:54.328223 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c597f9_e0f6_461a_aebf_cedb689b6772.slice/crio-5cf77d5763c9c8f818ee5e75691848f0b74c2be27dac698a344e2f01467ce198 WatchSource:0}: Error finding container 5cf77d5763c9c8f818ee5e75691848f0b74c2be27dac698a344e2f01467ce198: Status 404 returned error can't find the container with id 5cf77d5763c9c8f818ee5e75691848f0b74c2be27dac698a344e2f01467ce198 Oct 29 21:42:54.522573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:54.522541 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df8854f7d-rzpj7" event={"ID":"45c597f9-e0f6-461a-aebf-cedb689b6772","Type":"ContainerStarted","Data":"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc"} Oct 29 21:42:54.522749 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:54.522597 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df8854f7d-rzpj7" event={"ID":"45c597f9-e0f6-461a-aebf-cedb689b6772","Type":"ContainerStarted","Data":"5cf77d5763c9c8f818ee5e75691848f0b74c2be27dac698a344e2f01467ce198"} Oct 29 21:42:54.573243 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:54.573181 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6df8854f7d-rzpj7" podStartSLOduration=1.5731639020000001 podStartE2EDuration="1.573163902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:54.572649027 +0000 UTC m=+33.421125873" watchObservedRunningTime="2026-10-29 21:42:54.573163902 +0000 UTC m=+33.421640749" Oct 29 21:42:55.280530 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:55.280481 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:55.281122 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:55.280565 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:55.286780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:55.286745 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:42:55.531815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:42:55.531727 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:43:03.950781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:03.950575 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:43:03.951380 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:03.951357 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:43:03.956688 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:03.956661 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:43:04.566392 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:04.566367 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:43:04.642134 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:04.642104 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:43:06.852717 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:06.852674 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:43:06.853277 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:06.852747 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:43:16.431924 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:16.431863 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:43:26.858207 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:26.858177 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:43:26.861753 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:26.861735 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cb667f678-wvsbj" Oct 29 21:43:29.664003 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:29.663961 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c994468bf-jdtdd" podUID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" containerName="console" containerID="cri-o://36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3" gracePeriod=15 Oct 29 21:43:29.933959 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:29.933882 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:43:30.073165 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073118 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073165 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073170 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073204 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073230 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073258 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\" (UniqueName: \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073296 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073393 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\") pod \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\" (UID: \"fc1eb0cc-d288-431e-8888-4f12a3eeedec\") " Oct 29 21:43:30.073692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073530 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:30.073692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073593 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:30.073692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073612 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:30.073692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073647 4376 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1eb0cc-d288-431e-8888-4f12a3eeedec-tmp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.073869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.073847 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:30.075698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.075672 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:30.075797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.075693 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "kube-api-access-jqgpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:30.075797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.075693 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config" pod "fc1eb0cc-d288-431e-8888-4f12a3eeedec" (UID: "fc1eb0cc-d288-431e-8888-4f12a3eeedec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:30.174319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174283 4376 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.174319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174314 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgpc\" (UniqueName: \"kubernetes.io/projected/fc1eb0cc-d288-431e-8888-4f12a3eeedec-kube-api-access-jqgpc\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.174319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174324 4376 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.174555 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174334 4376 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-oauth-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.174555 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174343 4376 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-oauth-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.174555 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.174351 4376 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1eb0cc-d288-431e-8888-4f12a3eeedec-console-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:30.658819 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.658778 4376 generic.go:356] "Generic (PLEG): container finished" podID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" containerID="36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3" exitCode=0 Oct 29 21:43:30.659029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.658826 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c994468bf-jdtdd" Oct 29 21:43:30.659029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.658857 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c994468bf-jdtdd" event={"ID":"fc1eb0cc-d288-431e-8888-4f12a3eeedec","Type":"ContainerDied","Data":"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3"} Oct 29 21:43:30.659029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.658929 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c994468bf-jdtdd" event={"ID":"fc1eb0cc-d288-431e-8888-4f12a3eeedec","Type":"ContainerDied","Data":"b55dacf26f59ba306dc47ae1b56a72ee738a801a72bc6bb4407618e5679d73eb"} Oct 29 21:43:30.659029 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.658952 4376 scope.go:122] "RemoveContainer" containerID="36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3" Oct 29 21:43:30.667405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.667268 4376 scope.go:122] "RemoveContainer" containerID="36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3" Oct 29 21:43:30.667660 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:30.667543 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3\": container with ID starting with 36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3 not found: ID does not exist" containerID="36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3" Oct 29 21:43:30.667660 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.667569 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3"} err="failed to get container status \"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3\": rpc error: code = NotFound desc = could not find container \"36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3\": container with ID starting with 36d59f511254aa98fae72a010afe3db3e9a254cd1ef5ec9eab95edebed8222c3 not found: ID does not exist" Oct 29 21:43:30.713435 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.713403 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:43:30.725613 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:30.725589 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c994468bf-jdtdd"] Oct 29 21:43:31.269044 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:31.269002 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" path="/var/lib/kubelet/pods/fc1eb0cc-d288-431e-8888-4f12a3eeedec/volumes" Oct 29 21:43:46.441758 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:46.441706 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:43:53.863914 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.863866 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864159 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="alertmanager" containerID="cri-o://f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8" gracePeriod=120 Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864218 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-metric" containerID="cri-o://d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08" gracePeriod=120 Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864248 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-web" containerID="cri-o://b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65" gracePeriod=120 Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864285 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="config-reloader" containerID="cri-o://14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5" gracePeriod=120 Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864340 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="prom-label-proxy" containerID="cri-o://6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4" gracePeriod=120 Oct 29 21:43:53.864486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:53.864285 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy" containerID="cri-o://68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1" gracePeriod=120 Oct 29 21:43:54.750583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750549 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4" exitCode=0 Oct 29 21:43:54.750583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750575 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08" exitCode=0 Oct 29 21:43:54.750583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750582 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1" exitCode=0 Oct 29 21:43:54.750583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750588 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5" exitCode=0 Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750597 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8" exitCode=0 Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750616 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4"} Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750679 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08"} Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1"} Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750730 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5"} Oct 29 21:43:54.750853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:54.750752 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8"} Oct 29 21:43:55.098968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.098931 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.180968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.180932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.180968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.180973 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.180993 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181014 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181045 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181076 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181146 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181173 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181227 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181619 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181260 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181619 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181287 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181619 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181351 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\") pod \"fac8c644-70b8-46cc-b12d-182853a63661\" (UID: \"fac8c644-70b8-46cc-b12d-182853a63661\") " Oct 29 21:43:55.181773 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.181735 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:55.182492 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.182244 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:55.182918 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.182848 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:55.183899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.183864 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.184408 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.184378 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.184663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.184626 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.184883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.184847 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "kube-api-access-vwlgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:55.184883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.184852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.185097 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.185069 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:55.185314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.185298 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.185962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.185946 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:55.189145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.189045 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.198932 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.198747 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config" pod "fac8c644-70b8-46cc-b12d-182853a63661" (UID: "fac8c644-70b8-46cc-b12d-182853a63661"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:55.282882 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282858 4376 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282899 4376 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-config-out\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282928 4376 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282937 4376 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-main-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282947 4376 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282956 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlgk\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-kube-api-access-vwlgk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282964 4376 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fac8c644-70b8-46cc-b12d-182853a63661-tls-assets\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282972 4376 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-web-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282980 4376 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282988 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac8c644-70b8-46cc-b12d-182853a63661-metrics-client-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.282997 4376 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-cluster-tls-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.283005 4376 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fac8c644-70b8-46cc-b12d-182853a63661-alertmanager-main-db\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.283340 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.283013 4376 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fac8c644-70b8-46cc-b12d-182853a63661-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:55.757014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.756981 4376 generic.go:356] "Generic (PLEG): container finished" podID="fac8c644-70b8-46cc-b12d-182853a63661" containerID="b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65" exitCode=0 Oct 29 21:43:55.757192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.757077 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65"} Oct 29 21:43:55.757192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.757098 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.757270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.757194 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fac8c644-70b8-46cc-b12d-182853a63661","Type":"ContainerDied","Data":"5d2654b335a770d453e210ea120d2c9215b2f6e87b44bffb528c6d9a6f701dac"} Oct 29 21:43:55.757270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.757244 4376 scope.go:122] "RemoveContainer" containerID="6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4" Oct 29 21:43:55.764490 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.764470 4376 scope.go:122] "RemoveContainer" containerID="d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08" Oct 29 21:43:55.770983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.770964 4376 scope.go:122] "RemoveContainer" containerID="68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1" Oct 29 21:43:55.777348 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.777333 4376 scope.go:122] "RemoveContainer" containerID="b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65" Oct 29 21:43:55.783401 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.783384 4376 scope.go:122] "RemoveContainer" containerID="14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5" Oct 29 21:43:55.789742 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.789727 4376 scope.go:122] "RemoveContainer" containerID="f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8" Oct 29 21:43:55.795947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.795923 4376 scope.go:122] "RemoveContainer" containerID="b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235" Oct 29 21:43:55.802152 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802130 4376 scope.go:122] "RemoveContainer" containerID="6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4" Oct 29 21:43:55.802391 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.802372 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4\": container with ID starting with 6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4 not found: ID does not exist" containerID="6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4" Oct 29 21:43:55.802462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802404 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4"} err="failed to get container status \"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4\": rpc error: code = NotFound desc = could not find container \"6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4\": container with ID starting with 6757e15dde096aded9f305a8291a057ba2255f872ab91cbfcff52eea411003e4 not found: ID does not exist" Oct 29 21:43:55.802462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802428 4376 scope.go:122] "RemoveContainer" containerID="d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08" Oct 29 21:43:55.802679 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.802662 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08\": container with ID starting with d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08 not found: ID does not exist" containerID="d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08" Oct 29 21:43:55.802719 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802686 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08"} err="failed to get container status \"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08\": rpc error: code = NotFound desc = could not find container \"d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08\": container with ID starting with d3a449557d300b9e3e4fe888e4cf6caa1bbcdead98e25ac8609e29aefcdabd08 not found: ID does not exist" Oct 29 21:43:55.802719 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802701 4376 scope.go:122] "RemoveContainer" containerID="68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1" Oct 29 21:43:55.802946 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.802927 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1\": container with ID starting with 68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1 not found: ID does not exist" containerID="68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1" Oct 29 21:43:55.803014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802953 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1"} err="failed to get container status \"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1\": rpc error: code = NotFound desc = could not find container \"68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1\": container with ID starting with 68777746f9bfaa991487735a25e7acf786d7246870d552dd2fe51680cee412d1 not found: ID does not exist" Oct 29 21:43:55.803014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.802973 4376 scope.go:122] "RemoveContainer" containerID="b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65" Oct 29 21:43:55.803216 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.803199 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65\": container with ID starting with b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65 not found: ID does not exist" containerID="b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65" Oct 29 21:43:55.803253 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803221 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65"} err="failed to get container status \"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65\": rpc error: code = NotFound desc = could not find container \"b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65\": container with ID starting with b17ee86b35b3bab3fd0303e75c561a5fe539f44f78baccf0b4af8957e36c7e65 not found: ID does not exist" Oct 29 21:43:55.803253 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803234 4376 scope.go:122] "RemoveContainer" containerID="14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5" Oct 29 21:43:55.803441 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.803424 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5\": container with ID starting with 14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5 not found: ID does not exist" containerID="14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5" Oct 29 21:43:55.803501 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803448 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5"} err="failed to get container status \"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5\": rpc error: code = NotFound desc = could not find container \"14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5\": container with ID starting with 14335a40609c12f3bbb446b675d6ff4276ddde1fab7145d8c1cfc395aa4ffbf5 not found: ID does not exist" Oct 29 21:43:55.803501 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803468 4376 scope.go:122] "RemoveContainer" containerID="f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8" Oct 29 21:43:55.803678 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.803663 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8\": container with ID starting with f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8 not found: ID does not exist" containerID="f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8" Oct 29 21:43:55.803711 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803684 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8"} err="failed to get container status \"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8\": rpc error: code = NotFound desc = could not find container \"f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8\": container with ID starting with f3632d25f9405b5fa839a2890ec12a048ad085f8c51467535a0f52ea00ec1fe8 not found: ID does not exist" Oct 29 21:43:55.803711 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803698 4376 scope.go:122] "RemoveContainer" containerID="b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235" Oct 29 21:43:55.803880 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:43:55.803865 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235\": container with ID starting with b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235 not found: ID does not exist" containerID="b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235" Oct 29 21:43:55.803973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.803886 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235"} err="failed to get container status \"b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235\": rpc error: code = NotFound desc = could not find container \"b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235\": container with ID starting with b613411ddb9a38dcd6a916ee18ac4ea72b51e60d1585dcfdf78ddff92a68c235 not found: ID does not exist" Oct 29 21:43:55.830689 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.830666 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:55.843034 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.843014 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:55.881792 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.881766 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:55.882059 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882046 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-metric" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882060 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-metric" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882072 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="init-config-reloader" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882077 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="init-config-reloader" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882083 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="alertmanager" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882088 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="alertmanager" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882099 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="config-reloader" Oct 29 21:43:55.882109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882105 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="config-reloader" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882121 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882126 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882133 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" containerName="console" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882139 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" containerName="console" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882148 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="prom-label-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882152 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="prom-label-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882161 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-web" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882166 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-web" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882216 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882223 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1eb0cc-d288-431e-8888-4f12a3eeedec" containerName="console" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882229 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="prom-label-proxy" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882236 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-web" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882242 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="kube-rbac-proxy-metric" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882248 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="alertmanager" Oct 29 21:43:55.882299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.882253 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8c644-70b8-46cc-b12d-182853a63661" containerName="config-reloader" Oct 29 21:43:55.885670 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.885654 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.911031 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911009 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911037 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911064 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911079 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911093 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911079 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911105 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 21:43:55.911130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911113 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 21:43:55.911426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.911173 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-78gx5\"" Oct 29 21:43:55.924179 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.924151 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 21:43:55.925809 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.925788 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:55.987551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987525 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987558 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987574 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkl7t\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-kube-api-access-tkl7t\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987591 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-web-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987747 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-out\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987764 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987868 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987842 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987921 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987874 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987957 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987937 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.987986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987969 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:55.988017 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:55.987989 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088547 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088567 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkl7t\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-kube-api-access-tkl7t\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088600 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088589 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-web-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088633 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088655 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-out\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088678 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088703 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088744 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.088949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.088765 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.089522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.089292 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.089522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.089363 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.089522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.089368 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.089522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.089395 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.089772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.089590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.091176 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.091150 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.091695 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.091670 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092132 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092135 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092228 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-web-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092303 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092415 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092373 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092415 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092384 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-config-out\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.092517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.092503 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.093251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.093233 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.113952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.113933 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkl7t\" (UniqueName: \"kubernetes.io/projected/8eefb702-9336-4ab0-aa4d-8a584b1a1d4f-kube-api-access-tkl7t\") pod \"alertmanager-main-0\" (UID: \"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.202650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.202620 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 21:43:56.540637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.540602 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 21:43:56.543677 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:43:56.543649 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eefb702_9336_4ab0_aa4d_8a584b1a1d4f.slice/crio-3da3c5d016f13f10dec6ca1b3c996430b91d2c97afd88af0da745848258cf174 WatchSource:0}: Error finding container 3da3c5d016f13f10dec6ca1b3c996430b91d2c97afd88af0da745848258cf174: Status 404 returned error can't find the container with id 3da3c5d016f13f10dec6ca1b3c996430b91d2c97afd88af0da745848258cf174 Oct 29 21:43:56.761831 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.761798 4376 generic.go:356] "Generic (PLEG): container finished" podID="8eefb702-9336-4ab0-aa4d-8a584b1a1d4f" containerID="1ec42ed1ca453425c113f3c54ec76dff70eda44a5534770ccb2a187a9bf52ba0" exitCode=0 Oct 29 21:43:56.762027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.761876 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerDied","Data":"1ec42ed1ca453425c113f3c54ec76dff70eda44a5534770ccb2a187a9bf52ba0"} Oct 29 21:43:56.762027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:56.761973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"3da3c5d016f13f10dec6ca1b3c996430b91d2c97afd88af0da745848258cf174"} Oct 29 21:43:57.268812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.268781 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac8c644-70b8-46cc-b12d-182853a63661" path="/var/lib/kubelet/pods/fac8c644-70b8-46cc-b12d-182853a63661/volumes" Oct 29 21:43:57.772075 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772038 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"385d28d5cf76200a5628eaa8e6e9799e80af30ce89608e0e5af26a813de4f22e"} Oct 29 21:43:57.772252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772104 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"108c3d8e238d4db6d6daf2c4e9bb4c7144fb48cb311fa657055941fa32d1a3e2"} Oct 29 21:43:57.772252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772134 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"3ccfdd1ba9cc463ef3d2c3bc380c083f372eaed2936d9fefacf1e7bed76a3cd7"} Oct 29 21:43:57.772252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772162 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"e840c062326042219d493c46ed582986e445d0616001edd883737842ca328412"} Oct 29 21:43:57.772252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772182 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"8818455389c835d8f6418818ea986c986ba3d1fa2a31fe599b5760fef2412056"} Oct 29 21:43:57.772252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.772203 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8eefb702-9336-4ab0-aa4d-8a584b1a1d4f","Type":"ContainerStarted","Data":"83465c1687733ef28bf9caa30fbb7283dc6341fe353aa0c0e54210e2e8f7ce9d"} Oct 29 21:43:57.815967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.815936 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq"] Oct 29 21:43:57.818881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.818861 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.839250 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.839213 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.830482917 podStartE2EDuration="2.839201121s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.718204ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:43:57.838011556 +0000 UTC m=+96.686488420" watchObservedRunningTime="2026-10-29 21:43:57.839201121 +0000 UTC m=+96.687677965" Oct 29 21:43:57.850356 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.850331 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 29 21:43:57.850356 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.850331 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 29 21:43:57.850545 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.850358 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 29 21:43:57.850545 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.850363 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 29 21:43:57.850545 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.850402 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n55nc\"" Oct 29 21:43:57.860823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.860787 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq"] Oct 29 21:43:57.866533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.866511 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 29 21:43:57.872371 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.872339 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 29 21:43:57.905619 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.905592 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.905802 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.905780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf54l\" (UniqueName: \"kubernetes.io/projected/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-kube-api-access-sf54l\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.905877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.905829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.905957 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.905881 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-serving-certs-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.906015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.905997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-metrics-client-ca\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.906072 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.906030 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-federate-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.906304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.906286 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:57.906377 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:57.906326 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007566 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007614 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf54l\" (UniqueName: \"kubernetes.io/projected/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-kube-api-access-sf54l\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007655 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007688 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-serving-certs-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007709 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-metrics-client-ca\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.007764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007732 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-federate-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.008035 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007766 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.008035 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.007801 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.008611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.008581 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-serving-certs-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.008743 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.008584 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-metrics-client-ca\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.008807 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.008750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.010333 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.010309 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-federate-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.010531 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.010492 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-telemeter-client-tls\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.010596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.010495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.010639 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.010629 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-secret-telemeter-client\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.032398 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.032349 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf54l\" (UniqueName: \"kubernetes.io/projected/7c2b63ec-bb25-4889-a94f-473fc2a92cbb-kube-api-access-sf54l\") pod \"telemeter-client-7bc8cf59b7-2kzcq\" (UID: \"7c2b63ec-bb25-4889-a94f-473fc2a92cbb\") " pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.131889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.131832 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" Oct 29 21:43:58.470557 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.470520 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq"] Oct 29 21:43:58.473845 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:43:58.473811 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2b63ec_bb25_4889_a94f_473fc2a92cbb.slice/crio-81905e4bddf2ac6118561c74157b9389bda56e726e50c5cd4997f5cd957d7cd9 WatchSource:0}: Error finding container 81905e4bddf2ac6118561c74157b9389bda56e726e50c5cd4997f5cd957d7cd9: Status 404 returned error can't find the container with id 81905e4bddf2ac6118561c74157b9389bda56e726e50c5cd4997f5cd957d7cd9 Oct 29 21:43:58.777570 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:43:58.777487 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" event={"ID":"7c2b63ec-bb25-4889-a94f-473fc2a92cbb","Type":"ContainerStarted","Data":"81905e4bddf2ac6118561c74157b9389bda56e726e50c5cd4997f5cd957d7cd9"} Oct 29 21:44:00.786669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:00.786628 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" event={"ID":"7c2b63ec-bb25-4889-a94f-473fc2a92cbb","Type":"ContainerStarted","Data":"5b0c1ef720a5cb09ad61e180211cca7b2286b601b919c5b6c4e6496d5e417bcf"} Oct 29 21:44:00.787085 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:00.786720 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" event={"ID":"7c2b63ec-bb25-4889-a94f-473fc2a92cbb","Type":"ContainerStarted","Data":"88fa34f7fd9982d00c6bda43fd66493ce857343b4915b19431551d3d4bea6689"} Oct 29 21:44:00.787085 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:00.786739 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" event={"ID":"7c2b63ec-bb25-4889-a94f-473fc2a92cbb","Type":"ContainerStarted","Data":"4bac700654d513b89ca31adff13d0b7a812bb767c1dedacb35f58913ccba6b9d"} Oct 29 21:44:00.863236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:00.863181 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bc8cf59b7-2kzcq" podStartSLOduration=2.4887751590000002 podStartE2EDuration="3.863167272s" totalImagesPullingTime="1.374392113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:44:00.862895621 +0000 UTC m=+99.711372467" watchObservedRunningTime="2026-10-29 21:44:00.863167272 +0000 UTC m=+99.711644153" Oct 29 21:44:01.038627 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.038549 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:44:01.040757 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.040739 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.065160 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.065136 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:44:01.135016 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.134972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6q9\" (UniqueName: \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135195 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135035 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135195 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135114 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135195 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135147 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135195 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135172 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135274 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135310 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.135358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.135327 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235745 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235710 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235755 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235775 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235797 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6q9\" (UniqueName: \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235834 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235856 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.235948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.235916 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.236517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.236487 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.236612 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.236589 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.236781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.236764 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.236844 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.236790 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.237000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.236982 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.238318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.238303 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.238406 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.238327 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.261679 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.261643 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6q9\" (UniqueName: \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\") pod \"console-6d597d5c98-j5ftq\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.354755 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.354656 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:01.659268 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.659186 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:44:01.661898 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:44:01.661875 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82177d5d_7f94_4a6a_bb83_8547e80b53da.slice/crio-92dbe1d1603729d8ec606b120370022f30eb16a7556624b7f02823aa902c93c3 WatchSource:0}: Error finding container 92dbe1d1603729d8ec606b120370022f30eb16a7556624b7f02823aa902c93c3: Status 404 returned error can't find the container with id 92dbe1d1603729d8ec606b120370022f30eb16a7556624b7f02823aa902c93c3 Oct 29 21:44:01.792060 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.792023 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d597d5c98-j5ftq" event={"ID":"82177d5d-7f94-4a6a-bb83-8547e80b53da","Type":"ContainerStarted","Data":"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db"} Oct 29 21:44:01.792455 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.792100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d597d5c98-j5ftq" event={"ID":"82177d5d-7f94-4a6a-bb83-8547e80b53da","Type":"ContainerStarted","Data":"92dbe1d1603729d8ec606b120370022f30eb16a7556624b7f02823aa902c93c3"} Oct 29 21:44:01.838491 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:01.838434 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6d597d5c98-j5ftq" podStartSLOduration=0.838415938 podStartE2EDuration="838.415938ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:44:01.837941229 +0000 UTC m=+100.686418069" watchObservedRunningTime="2026-10-29 21:44:01.838415938 +0000 UTC m=+100.686892782" Oct 29 21:44:11.355292 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:11.355261 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:11.355746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:11.355316 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:11.360936 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:11.360896 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:11.829189 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:11.829159 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:44:11.911895 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:11.911855 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:44:16.432246 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:44:16.432208 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:44:36.933387 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:36.933285 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6df8854f7d-rzpj7" podUID="45c597f9-e0f6-461a-aebf-cedb689b6772" containerName="console" containerID="cri-o://64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc" gracePeriod=15 Oct 29 21:44:37.168893 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.168870 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:44:37.237260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237231 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\" (UniqueName: \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237280 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237302 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237368 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237391 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237427 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237723 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237455 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\" (UniqueName: \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\") pod \"45c597f9-e0f6-461a-aebf-cedb689b6772\" (UID: \"45c597f9-e0f6-461a-aebf-cedb689b6772\") " Oct 29 21:44:37.237723 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237537 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:44:37.237838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:37.237838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237725 4376 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45c597f9-e0f6-461a-aebf-cedb689b6772-tmp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.237961 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237829 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:37.238011 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.237914 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:37.238070 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.238045 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:37.239679 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.239652 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:44:37.239994 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.239968 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:44:37.239994 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.239986 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5" pod "45c597f9-e0f6-461a-aebf-cedb689b6772" (UID: "45c597f9-e0f6-461a-aebf-cedb689b6772"). InnerVolumeSpecName "kube-api-access-lhql5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:44:37.338360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338324 4376 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-oauth-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338353 4376 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-trusted-ca-bundle\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338362 4376 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c597f9-e0f6-461a-aebf-cedb689b6772-console-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338372 4376 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-console-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338380 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhql5\" (UniqueName: \"kubernetes.io/projected/45c597f9-e0f6-461a-aebf-cedb689b6772-kube-api-access-lhql5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338389 4376 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-oauth-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.338615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.338397 4376 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c597f9-e0f6-461a-aebf-cedb689b6772-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:37.920710 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.920674 4376 generic.go:356] "Generic (PLEG): container finished" podID="45c597f9-e0f6-461a-aebf-cedb689b6772" containerID="64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc" exitCode=0 Oct 29 21:44:37.920887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.920720 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df8854f7d-rzpj7" Oct 29 21:44:37.920887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.920752 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df8854f7d-rzpj7" event={"ID":"45c597f9-e0f6-461a-aebf-cedb689b6772","Type":"ContainerDied","Data":"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc"} Oct 29 21:44:37.920887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.920801 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df8854f7d-rzpj7" event={"ID":"45c597f9-e0f6-461a-aebf-cedb689b6772","Type":"ContainerDied","Data":"5cf77d5763c9c8f818ee5e75691848f0b74c2be27dac698a344e2f01467ce198"} Oct 29 21:44:37.920887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.920822 4376 scope.go:122] "RemoveContainer" containerID="64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc" Oct 29 21:44:37.928732 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.928711 4376 scope.go:122] "RemoveContainer" containerID="64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc" Oct 29 21:44:37.929070 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:44:37.929049 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc\": container with ID starting with 64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc not found: ID does not exist" containerID="64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc" Oct 29 21:44:37.929133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.929080 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc"} err="failed to get container status \"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc\": rpc error: code = NotFound desc = could not find container \"64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc\": container with ID starting with 64f62d5aea73789c2d9dec9462b369a1d3725c29633f65cce7500859997c27dc not found: ID does not exist" Oct 29 21:44:37.976790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.976752 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:44:37.986045 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:37.986014 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6df8854f7d-rzpj7"] Oct 29 21:44:39.269499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:44:39.269462 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c597f9-e0f6-461a-aebf-cedb689b6772" path="/var/lib/kubelet/pods/45c597f9-e0f6-461a-aebf-cedb689b6772/volumes" Oct 29 21:44:46.432145 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:44:46.432097 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:45:15.206893 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.206856 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d9b6fb66-57pvx"] Oct 29 21:45:15.207394 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.207179 4376 cpu_manager.go:414] "containerMap: removing container" podUID="45c597f9-e0f6-461a-aebf-cedb689b6772" containerName="console" Oct 29 21:45:15.207394 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.207191 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c597f9-e0f6-461a-aebf-cedb689b6772" containerName="console" Oct 29 21:45:15.207394 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.207243 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c597f9-e0f6-461a-aebf-cedb689b6772" containerName="console" Oct 29 21:45:15.210078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.210057 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.233084 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.233053 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d9b6fb66-57pvx"] Oct 29 21:45:15.340121 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340089 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-trusted-ca-bundle\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340133 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340151 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-oauth-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340236 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-service-ca\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340261 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-oauth-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340277 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-tmp\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.340533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.340376 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbh6\" (UniqueName: \"kubernetes.io/projected/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-kube-api-access-kvbh6\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441313 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441273 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-service-ca\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441327 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-oauth-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441351 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-tmp\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441383 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441406 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbh6\" (UniqueName: \"kubernetes.io/projected/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-kube-api-access-kvbh6\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441448 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-trusted-ca-bundle\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441779 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441525 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-oauth-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.441892 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.441872 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-tmp\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.442130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.442107 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-oauth-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.442297 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.442273 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.442375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.442329 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-trusted-ca-bundle\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.442375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.442358 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-service-ca\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.444038 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.444015 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-serving-cert\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.444179 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.444162 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-console-oauth-config\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.466936 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.466868 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbh6\" (UniqueName: \"kubernetes.io/projected/ac0eddce-c9cd-469c-8b86-37b80e4a5bca-kube-api-access-kvbh6\") pod \"console-57d9b6fb66-57pvx\" (UID: \"ac0eddce-c9cd-469c-8b86-37b80e4a5bca\") " pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.521962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.521926 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:15.823456 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:15.823382 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d9b6fb66-57pvx"] Oct 29 21:45:15.826580 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:45:15.826551 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0eddce_c9cd_469c_8b86_37b80e4a5bca.slice/crio-fab9d2e2c928c155ae9a51e765fb94becac6165419ad69e280acdfd4390f8578 WatchSource:0}: Error finding container fab9d2e2c928c155ae9a51e765fb94becac6165419ad69e280acdfd4390f8578: Status 404 returned error can't find the container with id fab9d2e2c928c155ae9a51e765fb94becac6165419ad69e280acdfd4390f8578 Oct 29 21:45:16.067727 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:16.067693 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d9b6fb66-57pvx" event={"ID":"ac0eddce-c9cd-469c-8b86-37b80e4a5bca","Type":"ContainerStarted","Data":"364edcfc7834eee00f5dfc7c724d34be83f1d59da70c5a5baf6c25defb5d4344"} Oct 29 21:45:16.067922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:16.067769 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d9b6fb66-57pvx" event={"ID":"ac0eddce-c9cd-469c-8b86-37b80e4a5bca","Type":"ContainerStarted","Data":"fab9d2e2c928c155ae9a51e765fb94becac6165419ad69e280acdfd4390f8578"} Oct 29 21:45:16.125308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:16.125191 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57d9b6fb66-57pvx" podStartSLOduration=1.125177703 podStartE2EDuration="1.125177703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:16.11160298 +0000 UTC m=+174.960079824" watchObservedRunningTime="2026-10-29 21:45:16.125177703 +0000 UTC m=+174.973654546" Oct 29 21:45:16.431559 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:45:16.431478 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:45:25.522448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:25.522412 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:25.522878 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:25.522475 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:25.526982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:25.526960 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:26.106582 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:26.106554 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d9b6fb66-57pvx" Oct 29 21:45:26.184332 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:26.184300 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:45:46.431819 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:45:46.431778 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:45:51.208701 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.208631 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6d597d5c98-j5ftq" podUID="82177d5d-7f94-4a6a-bb83-8547e80b53da" containerName="console" containerID="cri-o://c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db" gracePeriod=15 Oct 29 21:45:51.448188 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.448157 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:45:51.521582 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521582 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521582 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521565 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521856 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521599 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521856 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521634 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\" (UniqueName: \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521856 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521667 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\" (UniqueName: \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.521856 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521832 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.522096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521928 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\") pod \"82177d5d-7f94-4a6a-bb83-8547e80b53da\" (UID: \"82177d5d-7f94-4a6a-bb83-8547e80b53da\") " Oct 29 21:45:51.522096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.521962 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:45:51.522096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522007 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:45:51.522096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522017 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:45:51.522281 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522262 4376 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.522318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522287 4376 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82177d5d-7f94-4a6a-bb83-8547e80b53da-tmp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.522318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522301 4376 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.522385 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522363 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:45:51.522418 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.522397 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:45:51.523845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.523823 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:45:51.523954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.523859 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:45:51.523954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.523861 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9" pod "82177d5d-7f94-4a6a-bb83-8547e80b53da" (UID: "82177d5d-7f94-4a6a-bb83-8547e80b53da"). InnerVolumeSpecName "kube-api-access-qh6q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:45:51.623344 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.623309 4376 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-oauth-config\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.623344 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.623339 4376 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82177d5d-7f94-4a6a-bb83-8547e80b53da-console-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.623344 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.623350 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6q9\" (UniqueName: \"kubernetes.io/projected/82177d5d-7f94-4a6a-bb83-8547e80b53da-kube-api-access-qh6q9\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.623578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.623359 4376 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-oauth-serving-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:51.623578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:51.623369 4376 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82177d5d-7f94-4a6a-bb83-8547e80b53da-trusted-ca-bundle\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:45:52.196467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.196426 4376 generic.go:356] "Generic (PLEG): container finished" podID="82177d5d-7f94-4a6a-bb83-8547e80b53da" containerID="c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db" exitCode=0 Oct 29 21:45:52.196722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.196479 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d597d5c98-j5ftq" Oct 29 21:45:52.196722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.196509 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d597d5c98-j5ftq" event={"ID":"82177d5d-7f94-4a6a-bb83-8547e80b53da","Type":"ContainerDied","Data":"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db"} Oct 29 21:45:52.196722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.196559 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d597d5c98-j5ftq" event={"ID":"82177d5d-7f94-4a6a-bb83-8547e80b53da","Type":"ContainerDied","Data":"92dbe1d1603729d8ec606b120370022f30eb16a7556624b7f02823aa902c93c3"} Oct 29 21:45:52.196722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.196580 4376 scope.go:122] "RemoveContainer" containerID="c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db" Oct 29 21:45:52.205356 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.205336 4376 scope.go:122] "RemoveContainer" containerID="c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db" Oct 29 21:45:52.205643 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:45:52.205621 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db\": container with ID starting with c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db not found: ID does not exist" containerID="c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db" Oct 29 21:45:52.205706 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.205651 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db"} err="failed to get container status \"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db\": rpc error: code = NotFound desc = could not find container \"c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db\": container with ID starting with c23fc0dd6c81b06a7b9d1fa3c0c2c3fab32bdb49f1be0d28ba27c4fe6210e6db not found: ID does not exist" Oct 29 21:45:52.250934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.250891 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:45:52.263484 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:52.263445 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d597d5c98-j5ftq"] Oct 29 21:45:53.268974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:45:53.268943 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82177d5d-7f94-4a6a-bb83-8547e80b53da" path="/var/lib/kubelet/pods/82177d5d-7f94-4a6a-bb83-8547e80b53da/volumes" Oct 29 21:46:16.432678 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:46:16.432639 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:46:46.432643 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:46:46.432606 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:47:00.634067 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.634032 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml"] Oct 29 21:47:00.634481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.634339 4376 cpu_manager.go:414] "containerMap: removing container" podUID="82177d5d-7f94-4a6a-bb83-8547e80b53da" containerName="console" Oct 29 21:47:00.634481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.634350 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82177d5d-7f94-4a6a-bb83-8547e80b53da" containerName="console" Oct 29 21:47:00.634481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.634424 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="82177d5d-7f94-4a6a-bb83-8547e80b53da" containerName="console" Oct 29 21:47:00.637546 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.637530 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.665627 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.665596 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wmwc7\"" Oct 29 21:47:00.665818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.665650 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 21:47:00.665818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.665698 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 21:47:00.666862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.666836 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml"] Oct 29 21:47:00.673286 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.673262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.673404 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.673311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.673404 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.673332 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwls\" (UniqueName: \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.774418 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.774383 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.774603 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.774436 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.774603 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.774558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwls\" (UniqueName: \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.774784 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.774769 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.774864 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.774844 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.805520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.805485 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwls\" (UniqueName: \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:00.954305 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:00.954274 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:01.193026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.192995 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:47:01.197213 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.197194 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.213559 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.213493 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 21:47:01.213559 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.213545 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 21:47:01.213711 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.213575 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 21:47:01.225663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.225627 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 21:47:01.225663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.225643 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 21:47:01.225663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.225633 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 21:47:01.225928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.225689 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wt9dr\"" Oct 29 21:47:01.227578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.227554 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:47:01.269466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.269439 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml"] Oct 29 21:47:01.270462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:47:01.270429 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc5b117_6d7c_49ba_bf71_5b50b60e6d43.slice/crio-88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c WatchSource:0}: Error finding container 88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c: Status 404 returned error can't find the container with id 88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c Oct 29 21:47:01.279058 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279035 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279084 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42mjf\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279155 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279206 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279274 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279240 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279317 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279271 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.279368 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.279350 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380432 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380484 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42mjf\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380595 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380673 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380792 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.380894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.380839 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.381361 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.381338 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.382832 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.382803 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.382951 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.382839 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.383193 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.383175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.383255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.383218 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.406851 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.406830 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.407151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.407129 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42mjf\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\") pod \"istiod-openshift-gateway-7876bfcb74-zsg2d\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.438382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.438346 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" event={"ID":"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43","Type":"ContainerStarted","Data":"88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c"} Oct 29 21:47:01.509160 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.509066 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:01.831053 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:01.830977 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:47:01.837811 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:47:01.837776 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0365af2c_6e4e_4132_90a2_7a8761d4eeee.slice/crio-7c5eb790929aa25c2ba25b559a68587fcfd746e958a66aeae69eb1ca37f7f00f WatchSource:0}: Error finding container 7c5eb790929aa25c2ba25b559a68587fcfd746e958a66aeae69eb1ca37f7f00f: Status 404 returned error can't find the container with id 7c5eb790929aa25c2ba25b559a68587fcfd746e958a66aeae69eb1ca37f7f00f Oct 29 21:47:02.443486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:02.443445 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" event={"ID":"0365af2c-6e4e-4132-90a2-7a8761d4eeee","Type":"ContainerStarted","Data":"7c5eb790929aa25c2ba25b559a68587fcfd746e958a66aeae69eb1ca37f7f00f"} Oct 29 21:47:04.751149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:04.751108 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:47:04.751441 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:04.751175 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:47:05.461014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:05.460971 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" event={"ID":"0365af2c-6e4e-4132-90a2-7a8761d4eeee","Type":"ContainerStarted","Data":"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e"} Oct 29 21:47:05.544334 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:05.543209 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" podStartSLOduration=1.633024309 podStartE2EDuration="4.543185797s" totalImagesPullingTime="2.910161488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:05.529960215 +0000 UTC m=+284.378437100" watchObservedRunningTime="2026-10-29 21:47:05.543185797 +0000 UTC m=+284.391662643" Oct 29 21:47:06.466337 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:06.466296 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:06.468198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:06.468175 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:47:07.470945 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:07.470891 4376 generic.go:356] "Generic (PLEG): container finished" podID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerID="2a5c7658e7221ce87dbc5c542026c8bfffdf570f35380637119db53c3f46c5e4" exitCode=0 Oct 29 21:47:07.471485 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:07.471002 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" event={"ID":"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43","Type":"ContainerDied","Data":"2a5c7658e7221ce87dbc5c542026c8bfffdf570f35380637119db53c3f46c5e4"} Oct 29 21:47:09.479711 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:09.479681 4376 generic.go:356] "Generic (PLEG): container finished" podID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerID="2425f8662fa7fe80fc8476ab54e8329ecd931f1bd63e9e971db3eb79f03a219f" exitCode=0 Oct 29 21:47:09.480092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:09.479759 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" event={"ID":"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43","Type":"ContainerDied","Data":"2425f8662fa7fe80fc8476ab54e8329ecd931f1bd63e9e971db3eb79f03a219f"} Oct 29 21:47:16.436348 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:47:16.436262 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:47:16.509771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:16.509733 4376 generic.go:356] "Generic (PLEG): container finished" podID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerID="8a5bdf96c7949cc9d6ab20a5dbfbcef1d3f33c193ee521b8adb840fe8392c211" exitCode=0 Oct 29 21:47:16.509992 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:16.509811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" event={"ID":"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43","Type":"ContainerDied","Data":"8a5bdf96c7949cc9d6ab20a5dbfbcef1d3f33c193ee521b8adb840fe8392c211"} Oct 29 21:47:17.639390 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.639360 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:17.736338 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.736294 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\" (UniqueName: \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\") pod \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " Oct 29 21:47:17.736541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.736349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\") pod \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " Oct 29 21:47:17.736541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.736395 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\") pod \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\" (UID: \"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43\") " Oct 29 21:47:17.736986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.736957 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle" pod "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" (UID: "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:17.738612 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.738582 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls" pod "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" (UID: "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43"). InnerVolumeSpecName "kube-api-access-shwls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:17.740506 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.740484 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util" pod "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" (UID: "3bc5b117-6d7c-49ba-bf71-5b50b60e6d43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:17.837575 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.837484 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwls\" (UniqueName: \"kubernetes.io/projected/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-kube-api-access-shwls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:47:17.837575 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.837521 4376 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-bundle\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:47:17.837575 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:17.837532 4376 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc5b117-6d7c-49ba-bf71-5b50b60e6d43-util\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:47:18.518757 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:18.518715 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" event={"ID":"3bc5b117-6d7c-49ba-bf71-5b50b60e6d43","Type":"ContainerDied","Data":"88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c"} Oct 29 21:47:18.518952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:18.518771 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2zkml" Oct 29 21:47:18.518952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:18.518774 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e2c4321663c37cb9404799160e46fe465e3f49bb58954ed4cbbec6047bc79c" Oct 29 21:47:46.432863 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:47:46.432779 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:47:50.669565 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669531 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27"] Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669851 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="util" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669861 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="util" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669886 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="pull" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669891 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="pull" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669915 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="extract" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669921 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="extract" Oct 29 21:47:50.669984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.669979 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc5b117-6d7c-49ba-bf71-5b50b60e6d43" containerName="extract" Oct 29 21:47:50.672019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.671994 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.709477 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.709442 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27"] Oct 29 21:47:50.720538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720486 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.720694 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720614 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.720741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720699 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.720741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720735 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.720807 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720772 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbvd\" (UniqueName: \"kubernetes.io/projected/d28db487-507e-47d2-9463-aea64139d885-kube-api-access-vqbvd\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.720852 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.720837 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.741053 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741025 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 21:47:50.741328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741305 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 21:47:50.741402 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741389 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 21:47:50.741541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741522 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 21:47:50.741665 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741647 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-flksv\"" Oct 29 21:47:50.741818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.741739 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 21:47:50.822303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822318 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822340 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822368 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822393 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822422 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbvd\" (UniqueName: \"kubernetes.io/projected/d28db487-507e-47d2-9463-aea64139d885-kube-api-access-vqbvd\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.822783 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.822747 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.824755 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.824725 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28db487-507e-47d2-9463-aea64139d885-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.824870 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.824799 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.824964 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.824949 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.825005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.824985 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d28db487-507e-47d2-9463-aea64139d885-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.855993 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.855954 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbvd\" (UniqueName: \"kubernetes.io/projected/d28db487-507e-47d2-9463-aea64139d885-kube-api-access-vqbvd\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-n9g27\" (UID: \"d28db487-507e-47d2-9463-aea64139d885\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:50.984811 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:50.984777 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:51.364024 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:51.363996 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27"] Oct 29 21:47:51.366155 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:47:51.366125 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28db487_507e_47d2_9463_aea64139d885.slice/crio-28f3a86d12db65cae293f565cd331116c18645af0efe7a8a5b1fbec6e453ac21 WatchSource:0}: Error finding container 28f3a86d12db65cae293f565cd331116c18645af0efe7a8a5b1fbec6e453ac21: Status 404 returned error can't find the container with id 28f3a86d12db65cae293f565cd331116c18645af0efe7a8a5b1fbec6e453ac21 Oct 29 21:47:51.367745 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:51.367727 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:47:51.643501 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:51.643414 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" event={"ID":"d28db487-507e-47d2-9463-aea64139d885","Type":"ContainerStarted","Data":"28f3a86d12db65cae293f565cd331116c18645af0efe7a8a5b1fbec6e453ac21"} Oct 29 21:47:54.658638 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:54.658600 4376 generic.go:356] "Generic (PLEG): container finished" podID="d28db487-507e-47d2-9463-aea64139d885" containerID="f2bf0400bdf5e5fd6017769f71ad1698fc1c7f94796e72d8f672cbb32f63be34" exitCode=0 Oct 29 21:47:54.659059 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:54.658660 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" event={"ID":"d28db487-507e-47d2-9463-aea64139d885","Type":"ContainerDied","Data":"f2bf0400bdf5e5fd6017769f71ad1698fc1c7f94796e72d8f672cbb32f63be34"} Oct 29 21:47:55.666625 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:55.666577 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" event={"ID":"d28db487-507e-47d2-9463-aea64139d885","Type":"ContainerStarted","Data":"415bbf3a15ef80f348d1b6ad197b7faf18d4a7d24d94f986b87aff8192566168"} Oct 29 21:47:55.667068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:55.666752 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:47:55.788950 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:47:55.788881 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" podStartSLOduration=2.254090388 podStartE2EDuration="5.788863451s" totalImagesPullingTime="3.116562948s" totalInitContainerRuntime="418.210115ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:55.77553358 +0000 UTC m=+334.624010426" watchObservedRunningTime="2026-10-29 21:47:55.788863451 +0000 UTC m=+334.637340296" Oct 29 21:48:06.673736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:06.673704 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-n9g27" Oct 29 21:48:16.432516 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:48:16.432479 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:48:27.448835 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.448772 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-mls86"] Oct 29 21:48:27.490119 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.490081 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-mls86"] Oct 29 21:48:27.490307 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.490162 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.519107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.519070 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5v6tw\"" Oct 29 21:48:27.519107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.519070 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 21:48:27.519107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.519082 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 21:48:27.553623 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.553586 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tls-certs\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.553828 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.553656 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tmp\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.553828 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.553766 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95ns\" (UniqueName: \"kubernetes.io/projected/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-kube-api-access-z95ns\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.655039 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.654995 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tls-certs\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.655249 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.655092 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tmp\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.655249 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.655123 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95ns\" (UniqueName: \"kubernetes.io/projected/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-kube-api-access-z95ns\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.658027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.658000 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tmp\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.658287 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.658256 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-tls-certs\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.686206 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.686176 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95ns\" (UniqueName: \"kubernetes.io/projected/0d14731c-01a7-4c2d-8308-d16d6df7cb5c-kube-api-access-z95ns\") pod \"kube-auth-proxy-698579f7b7-mls86\" (UID: \"0d14731c-01a7-4c2d-8308-d16d6df7cb5c\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:27.814101 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:27.814067 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" Oct 29 21:48:28.166648 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.166504 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-mls86"] Oct 29 21:48:28.169373 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:48:28.169343 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d14731c_01a7_4c2d_8308_d16d6df7cb5c.slice/crio-ba50e5f1e516892da6282fe9dbcb714fe99c8b268a8d0a5f982f531d27051aff WatchSource:0}: Error finding container ba50e5f1e516892da6282fe9dbcb714fe99c8b268a8d0a5f982f531d27051aff: Status 404 returned error can't find the container with id ba50e5f1e516892da6282fe9dbcb714fe99c8b268a8d0a5f982f531d27051aff Oct 29 21:48:28.499151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.499116 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:28.501863 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.501846 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.518381 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.518350 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 21:48:28.530132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.530099 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 21:48:28.530306 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.530136 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 21:48:28.530378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.530339 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jjjl6\"" Oct 29 21:48:28.532127 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.532105 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:28.567936 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.567891 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.568132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.568014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdb8\" (UniqueName: \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.645599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.645566 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:28.669503 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.669466 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.669705 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.669514 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdb8\" (UniqueName: \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.670203 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.670178 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.696926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.696876 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdb8\" (UniqueName: \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\") pod \"limitador-limitador-74dfcfb894-n4d4v\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:28.804016 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.803897 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" event={"ID":"0d14731c-01a7-4c2d-8308-d16d6df7cb5c","Type":"ContainerStarted","Data":"ba50e5f1e516892da6282fe9dbcb714fe99c8b268a8d0a5f982f531d27051aff"} Oct 29 21:48:28.814200 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:28.814164 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:29.139201 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:29.139167 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:29.142009 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:48:29.141971 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1b6592_88fc_45e1_8cc7_b04834d24f10.slice/crio-c1c5ba3db18336296752a9c0a2d62c82bb61fe3fe1940ad5e5b9ecd3071aa792 WatchSource:0}: Error finding container c1c5ba3db18336296752a9c0a2d62c82bb61fe3fe1940ad5e5b9ecd3071aa792: Status 404 returned error can't find the container with id c1c5ba3db18336296752a9c0a2d62c82bb61fe3fe1940ad5e5b9ecd3071aa792 Oct 29 21:48:29.809986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:29.809946 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" event={"ID":"2d1b6592-88fc-45e1-8cc7-b04834d24f10","Type":"ContainerStarted","Data":"c1c5ba3db18336296752a9c0a2d62c82bb61fe3fe1940ad5e5b9ecd3071aa792"} Oct 29 21:48:30.817239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:30.817193 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" event={"ID":"0d14731c-01a7-4c2d-8308-d16d6df7cb5c","Type":"ContainerStarted","Data":"8d46449aa448a408d8fe059e8fbd1590c888d2a19ffdd0b39edd931a7135bc03"} Oct 29 21:48:30.860111 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:30.860038 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-698579f7b7-mls86" podStartSLOduration=1.315383584 podStartE2EDuration="3.860015093s" totalImagesPullingTime="2.544631509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:30.858520665 +0000 UTC m=+369.706997510" watchObservedRunningTime="2026-10-29 21:48:30.860015093 +0000 UTC m=+369.708491938" Oct 29 21:48:31.730134 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.730098 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7"] Oct 29 21:48:31.733502 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.733479 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.749954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.749896 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-j2p9r\"" Oct 29 21:48:31.761846 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.761810 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 21:48:31.761846 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.761813 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 21:48:31.787760 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.787724 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 29 21:48:31.790036 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.790008 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7"] Oct 29 21:48:31.809096 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.809063 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p248f\" (UniqueName: \"kubernetes.io/projected/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-kube-api-access-p248f\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.809233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.809118 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.809233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.809141 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.809345 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.809248 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.910446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.910401 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p248f\" (UniqueName: \"kubernetes.io/projected/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-kube-api-access-p248f\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.910974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.910491 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.910974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.910517 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.910974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.910542 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.911142 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.910998 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.911142 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.911073 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.911514 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.911494 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:31.936047 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:31.935964 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p248f\" (UniqueName: \"kubernetes.io/projected/f387d9bd-6ff0-4295-9da1-399a1e2d6b1f-kube-api-access-p248f\") pod \"kserve-module-controller-manager-7566cf5fc8-45zz7\" (UID: \"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:32.046759 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.046672 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:32.368092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.367958 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7"] Oct 29 21:48:32.374276 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:48:32.374066 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf387d9bd_6ff0_4295_9da1_399a1e2d6b1f.slice/crio-89f288702252e4ccf972bb74a0b19e7502fecdfaf2ebc5407a178e171b195164 WatchSource:0}: Error finding container 89f288702252e4ccf972bb74a0b19e7502fecdfaf2ebc5407a178e171b195164: Status 404 returned error can't find the container with id 89f288702252e4ccf972bb74a0b19e7502fecdfaf2ebc5407a178e171b195164 Oct 29 21:48:32.826233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.826189 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" event={"ID":"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f","Type":"ContainerStarted","Data":"89f288702252e4ccf972bb74a0b19e7502fecdfaf2ebc5407a178e171b195164"} Oct 29 21:48:32.827702 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.827675 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" event={"ID":"2d1b6592-88fc-45e1-8cc7-b04834d24f10","Type":"ContainerStarted","Data":"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2"} Oct 29 21:48:32.827864 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.827807 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:32.871042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:32.870979 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" podStartSLOduration=2.219931995 podStartE2EDuration="4.870965664s" totalImagesPullingTime="2.651033669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:32.87035338 +0000 UTC m=+371.718830225" watchObservedRunningTime="2026-10-29 21:48:32.870965664 +0000 UTC m=+371.719442557" Oct 29 21:48:34.843228 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:34.843183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" event={"ID":"f387d9bd-6ff0-4295-9da1-399a1e2d6b1f","Type":"ContainerStarted","Data":"970b8cc3571046ff341d28741d0652df13dfd28e9d77d2f9519b39182e123f86"} Oct 29 21:48:34.843663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:34.843289 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:34.902893 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:34.902829 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" podStartSLOduration=2.441088963 podStartE2EDuration="3.902813148s" totalImagesPullingTime="1.461724185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:34.901232872 +0000 UTC m=+373.749709719" watchObservedRunningTime="2026-10-29 21:48:34.902813148 +0000 UTC m=+373.751289991" Oct 29 21:48:43.641527 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:43.641493 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:43.641957 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:43.641690 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" podUID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" containerName="limitador" containerID="cri-o://0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2" gracePeriod=30 Oct 29 21:48:43.642344 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:43.642318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:44.576518 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.576493 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:44.730569 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.730457 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\" (UniqueName: \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\") pod \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " Oct 29 21:48:44.731092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.730614 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\" (UniqueName: \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\") pod \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\" (UID: \"2d1b6592-88fc-45e1-8cc7-b04834d24f10\") " Oct 29 21:48:44.731092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.730819 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file" pod "2d1b6592-88fc-45e1-8cc7-b04834d24f10" (UID: "2d1b6592-88fc-45e1-8cc7-b04834d24f10"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:48:44.731092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.730948 4376 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d1b6592-88fc-45e1-8cc7-b04834d24f10-config-file\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:48:44.733373 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.733341 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8" pod "2d1b6592-88fc-45e1-8cc7-b04834d24f10" (UID: "2d1b6592-88fc-45e1-8cc7-b04834d24f10"). InnerVolumeSpecName "kube-api-access-5bdb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:48:44.831996 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.831949 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdb8\" (UniqueName: \"kubernetes.io/projected/2d1b6592-88fc-45e1-8cc7-b04834d24f10-kube-api-access-5bdb8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:48:44.884679 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.884645 4376 generic.go:356] "Generic (PLEG): container finished" podID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" containerID="0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2" exitCode=0 Oct 29 21:48:44.884862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.884693 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" Oct 29 21:48:44.884862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.884730 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" event={"ID":"2d1b6592-88fc-45e1-8cc7-b04834d24f10","Type":"ContainerDied","Data":"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2"} Oct 29 21:48:44.884862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.884775 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-n4d4v" event={"ID":"2d1b6592-88fc-45e1-8cc7-b04834d24f10","Type":"ContainerDied","Data":"c1c5ba3db18336296752a9c0a2d62c82bb61fe3fe1940ad5e5b9ecd3071aa792"} Oct 29 21:48:44.884862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.884800 4376 scope.go:122] "RemoveContainer" containerID="0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2" Oct 29 21:48:44.893745 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.893725 4376 scope.go:122] "RemoveContainer" containerID="0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2" Oct 29 21:48:44.894132 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:48:44.894109 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2\": container with ID starting with 0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2 not found: ID does not exist" containerID="0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2" Oct 29 21:48:44.894196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.894143 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2"} err="failed to get container status \"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2\": rpc error: code = NotFound desc = could not find container \"0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2\": container with ID starting with 0719e999075b7518f7b4120272b924a98fc84dd0e332d52243528ae318ee96c2 not found: ID does not exist" Oct 29 21:48:44.939145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.939108 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:44.951637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:44.951600 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-n4d4v"] Oct 29 21:48:45.269359 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:45.269327 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" path="/var/lib/kubelet/pods/2d1b6592-88fc-45e1-8cc7-b04834d24f10/volumes" Oct 29 21:48:46.434002 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:48:46.433966 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:48:49.908812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.908777 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2"] Oct 29 21:48:49.909236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.909103 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" containerName="limitador" Oct 29 21:48:49.909236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.909113 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" containerName="limitador" Oct 29 21:48:49.909236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.909187 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1b6592-88fc-45e1-8cc7-b04834d24f10" containerName="limitador" Oct 29 21:48:49.913436 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.913416 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:49.933802 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.933768 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 29 21:48:49.933802 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.933769 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ktskz\"" Oct 29 21:48:49.934023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.933878 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2"] Oct 29 21:48:49.934023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.933966 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:48:49.934270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:49.934257 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:48:50.081888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.081846 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f5dc256-d231-44dc-97ef-43c4748be472-data\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.082099 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.081895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7rp\" (UniqueName: \"kubernetes.io/projected/4f5dc256-d231-44dc-97ef-43c4748be472-kube-api-access-dv7rp\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.182623 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.182530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f5dc256-d231-44dc-97ef-43c4748be472-data\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.182623 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.182593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7rp\" (UniqueName: \"kubernetes.io/projected/4f5dc256-d231-44dc-97ef-43c4748be472-kube-api-access-dv7rp\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.182949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.182931 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f5dc256-d231-44dc-97ef-43c4748be472-data\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.209943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.209891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7rp\" (UniqueName: \"kubernetes.io/projected/4f5dc256-d231-44dc-97ef-43c4748be472-kube-api-access-dv7rp\") pod \"postgres-7cdbf6dc44-2dxs2\" (UID: \"4f5dc256-d231-44dc-97ef-43c4748be472\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.226288 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.226254 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:50.528355 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.528222 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2"] Oct 29 21:48:50.531047 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:48:50.531015 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5dc256_d231_44dc_97ef_43c4748be472.slice/crio-546693630d8e208d191300178fd3b3f552a275272fbcc6e3ce46c14fa3b0c2e2 WatchSource:0}: Error finding container 546693630d8e208d191300178fd3b3f552a275272fbcc6e3ce46c14fa3b0c2e2: Status 404 returned error can't find the container with id 546693630d8e208d191300178fd3b3f552a275272fbcc6e3ce46c14fa3b0c2e2 Oct 29 21:48:50.913063 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:50.912976 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" event={"ID":"4f5dc256-d231-44dc-97ef-43c4748be472","Type":"ContainerStarted","Data":"546693630d8e208d191300178fd3b3f552a275272fbcc6e3ce46c14fa3b0c2e2"} Oct 29 21:48:55.848736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:55.848690 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-45zz7" Oct 29 21:48:55.937746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:55.937702 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" event={"ID":"4f5dc256-d231-44dc-97ef-43c4748be472","Type":"ContainerStarted","Data":"28c01ad5ede54d312990fee88c8dedd146d84bc75e5cf88d50f33616591e59a9"} Oct 29 21:48:55.937926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:55.937785 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:48:55.980358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:48:55.980307 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" podStartSLOduration=2.343726835 podStartE2EDuration="6.980290834s" totalImagesPullingTime="4.636563999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:55.979180798 +0000 UTC m=+394.827657654" watchObservedRunningTime="2026-10-29 21:48:55.980290834 +0000 UTC m=+394.828767680" Oct 29 21:49:01.977450 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:01.977402 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2dxs2" Oct 29 21:49:14.397539 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.397460 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr"] Oct 29 21:49:14.413791 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.413759 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.421639 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.421612 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr"] Oct 29 21:49:14.432015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.431978 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-vd668\"" Oct 29 21:49:14.432015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.431997 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 29 21:49:14.432015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.431986 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 29 21:49:14.432200 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.431986 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 29 21:49:14.480756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.480721 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7966cd6d-a377-4b75-b61e-d74cc0dd5021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.480756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.480757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqkv\" (UniqueName: \"kubernetes.io/projected/7966cd6d-a377-4b75-b61e-d74cc0dd5021-kube-api-access-9lqkv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.480999 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.480896 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.480999 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.480955 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.581967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.581927 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.582160 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.582025 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7966cd6d-a377-4b75-b61e-d74cc0dd5021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.582160 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.582052 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqkv\" (UniqueName: \"kubernetes.io/projected/7966cd6d-a377-4b75-b61e-d74cc0dd5021-kube-api-access-9lqkv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.582295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.582207 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.582662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.582631 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.582987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.582968 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7966cd6d-a377-4b75-b61e-d74cc0dd5021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.584398 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.584378 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7966cd6d-a377-4b75-b61e-d74cc0dd5021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.606899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.606874 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqkv\" (UniqueName: \"kubernetes.io/projected/7966cd6d-a377-4b75-b61e-d74cc0dd5021-kube-api-access-9lqkv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-79xrr\" (UID: \"7966cd6d-a377-4b75-b61e-d74cc0dd5021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:14.729564 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:14.729529 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:15.031043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:15.030953 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr"] Oct 29 21:49:16.020536 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:16.020493 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" event={"ID":"7966cd6d-a377-4b75-b61e-d74cc0dd5021","Type":"ContainerStarted","Data":"95968256ed187974bf5d03f95f6a2da6b780e493e1f3aadd54450d2b5d7187ff"} Oct 29 21:49:16.449675 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:49:16.449518 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:49:26.071767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:26.071728 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" event={"ID":"7966cd6d-a377-4b75-b61e-d74cc0dd5021","Type":"ContainerStarted","Data":"9ec356e1fee2b19dd26601d055934a9ac8cb76f06c939d70a3a3951fe0b1b656"} Oct 29 21:49:26.072228 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:26.071794 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:26.114830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:26.114765 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" podStartSLOduration=1.795973 podStartE2EDuration="12.114746981s" totalImagesPullingTime="10.318773981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:26.114268207 +0000 UTC m=+424.962745052" watchObservedRunningTime="2026-10-29 21:49:26.114746981 +0000 UTC m=+424.963223827" Oct 29 21:49:27.448449 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.448417 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 21:49:27.452000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.451976 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.480054 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.479997 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 21:49:27.517671 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517634 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.517671 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517679 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.517880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517700 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.517880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.517880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517837 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.518024 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517938 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.518024 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.517960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2f6\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.618855 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.618820 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.618855 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.618869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.618893 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.618947 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.619062 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619296 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.619165 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619296 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.619198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2f6\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.619927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.619863 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.621404 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.621381 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.621532 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.621459 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.621532 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.621480 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.621627 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.621548 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.645206 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.645172 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.645341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.645219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2f6\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mn5ld\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:27.764087 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:27.764045 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:28.085241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:28.085217 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 21:49:28.087459 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:49:28.087426 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b0c9d7_f5e3_408a_ac68_92b6a9a8387c.slice/crio-2e1f37fc564b0f51f83c5164cea60837f7991f3540fedeb166e6202c6c4c4291 WatchSource:0}: Error finding container 2e1f37fc564b0f51f83c5164cea60837f7991f3540fedeb166e6202c6c4c4291: Status 404 returned error can't find the container with id 2e1f37fc564b0f51f83c5164cea60837f7991f3540fedeb166e6202c6c4c4291 Oct 29 21:49:28.089495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:28.089463 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:49:28.089588 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:28.089524 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:49:29.085880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.085839 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" event={"ID":"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c","Type":"ContainerStarted","Data":"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d"} Oct 29 21:49:29.086352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.085894 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" event={"ID":"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c","Type":"ContainerStarted","Data":"2e1f37fc564b0f51f83c5164cea60837f7991f3540fedeb166e6202c6c4c4291"} Oct 29 21:49:29.086352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.085997 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:29.087776 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.087754 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 21:49:29.160615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.160563 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" podStartSLOduration=2.160550094 podStartE2EDuration="2.160550094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:29.147142826 +0000 UTC m=+427.995619672" watchObservedRunningTime="2026-10-29 21:49:29.160550094 +0000 UTC m=+428.009026936" Oct 29 21:49:29.236463 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.236423 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:49:29.236694 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.236668 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" podUID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" containerName="discovery" containerID="cri-o://0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e" gracePeriod=30 Oct 29 21:49:29.484351 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.484327 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:49:29.637889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637794 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.637889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637836 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.637889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637883 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.638226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637925 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.638226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637956 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.638226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.637998 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.638226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.638048 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\") pod \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\" (UID: \"0365af2c-6e4e-4132-90a2-7a8761d4eeee\") " Oct 29 21:49:29.638720 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.638685 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:49:29.640867 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640839 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "kube-api-access-42mjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:29.640867 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640848 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:49:29.641037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640833 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:29.641037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640873 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:29.641037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:29.641037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.640874 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token" pod "0365af2c-6e4e-4132-90a2-7a8761d4eeee" (UID: "0365af2c-6e4e-4132-90a2-7a8761d4eeee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:29.739536 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739491 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-dns-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739536 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739535 4376 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0365af2c-6e4e-4132-90a2-7a8761d4eeee-local-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739747 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739548 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-csr-ca-configmap\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739747 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739564 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739747 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739576 4376 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-cacerts\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739747 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739587 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42mjf\" (UniqueName: \"kubernetes.io/projected/0365af2c-6e4e-4132-90a2-7a8761d4eeee-kube-api-access-42mjf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:29.739747 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:29.739598 4376 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0365af2c-6e4e-4132-90a2-7a8761d4eeee-istio-kubeconfig\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:30.091248 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.091215 4376 generic.go:356] "Generic (PLEG): container finished" podID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" containerID="0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e" exitCode=0 Oct 29 21:49:30.091644 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.091263 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" Oct 29 21:49:30.091644 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.091301 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" event={"ID":"0365af2c-6e4e-4132-90a2-7a8761d4eeee","Type":"ContainerDied","Data":"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e"} Oct 29 21:49:30.091644 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.091353 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d" event={"ID":"0365af2c-6e4e-4132-90a2-7a8761d4eeee","Type":"ContainerDied","Data":"7c5eb790929aa25c2ba25b559a68587fcfd746e958a66aeae69eb1ca37f7f00f"} Oct 29 21:49:30.091644 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.091372 4376 scope.go:122] "RemoveContainer" containerID="0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e" Oct 29 21:49:30.100631 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.100608 4376 scope.go:122] "RemoveContainer" containerID="0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e" Oct 29 21:49:30.100896 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:49:30.100876 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e\": container with ID starting with 0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e not found: ID does not exist" containerID="0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e" Oct 29 21:49:30.100968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.100916 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e"} err="failed to get container status \"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e\": rpc error: code = NotFound desc = could not find container \"0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e\": container with ID starting with 0450b2f783fb80098963c47ef7568a998bbf90af0be01fae41d7bf0b9983415e not found: ID does not exist" Oct 29 21:49:30.162168 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.162128 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:49:30.174331 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:30.174301 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zsg2d"] Oct 29 21:49:31.272251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:31.272213 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" path="/var/lib/kubelet/pods/0365af2c-6e4e-4132-90a2-7a8761d4eeee/volumes" Oct 29 21:49:37.081621 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:37.081591 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-79xrr" Oct 29 21:49:45.664611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.664575 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:49:45.665034 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.664972 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" containerName="discovery" Oct 29 21:49:45.665034 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.664985 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" containerName="discovery" Oct 29 21:49:45.665105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.665044 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0365af2c-6e4e-4132-90a2-7a8761d4eeee" containerName="discovery" Oct 29 21:49:45.699683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.699645 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:49:45.699859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.699797 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.728279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.728244 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8hhr\"" Oct 29 21:49:45.728466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.728332 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 21:49:45.728466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.728413 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 21:49:45.892643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.892607 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.892840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.892654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.892840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.892683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktr2\" (UniqueName: \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.892951 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.892827 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.993669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.993630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.993669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.993673 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.993894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.993713 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktr2\" (UniqueName: \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.993894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.993742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.994480 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.994452 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.996134 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.996114 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:45.996240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:45.996222 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:46.018674 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:46.018630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktr2\" (UniqueName: \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\") pod \"maas-api-74b45cb84d-5g8bj\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:46.312712 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:46.312613 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:46.434084 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:49:46.434031 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:49:46.614528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:46.614433 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:49:46.618135 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:49:46.618097 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faafa00_c5c0_4619_ba68_7a1a4a71893c.slice/crio-de2b47b3a77d3066032bd1952a56ea640b0985c80bc1c1fbbb35486f4ae517c6 WatchSource:0}: Error finding container de2b47b3a77d3066032bd1952a56ea640b0985c80bc1c1fbbb35486f4ae517c6: Status 404 returned error can't find the container with id de2b47b3a77d3066032bd1952a56ea640b0985c80bc1c1fbbb35486f4ae517c6 Oct 29 21:49:47.167491 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:47.167449 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" event={"ID":"9faafa00-c5c0-4619-ba68-7a1a4a71893c","Type":"ContainerStarted","Data":"de2b47b3a77d3066032bd1952a56ea640b0985c80bc1c1fbbb35486f4ae517c6"} Oct 29 21:49:49.178728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:49.178689 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" event={"ID":"9faafa00-c5c0-4619-ba68-7a1a4a71893c","Type":"ContainerStarted","Data":"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795"} Oct 29 21:49:49.179247 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:49.178778 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:49.222067 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:49.221952 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" podStartSLOduration=1.881227232 podStartE2EDuration="4.221938409s" totalImagesPullingTime="2.340711177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:49.221485205 +0000 UTC m=+448.069962050" watchObservedRunningTime="2026-10-29 21:49:49.221938409 +0000 UTC m=+448.070415253" Oct 29 21:49:51.441316 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.441280 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 21:49:51.443805 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.443781 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.460547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.460517 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4b7xt\"" Oct 29 21:49:51.474369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.474334 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 21:49:51.544606 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544568 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.544790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544613 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.544790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544636 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.544790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544700 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.544889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544801 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgprv\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.544889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544828 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.545002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544952 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.545002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.545002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.544991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646083 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646143 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgprv\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646181 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646238 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646255 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646271 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646298 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646672 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646335 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646672 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646361 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646670 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646729 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646774 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.646972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.646954 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.647239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.647220 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.649235 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.649205 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.649398 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.649383 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.671737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.671696 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.671870 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.671811 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgprv\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\") pod \"maas-default-gateway-openshift-default-6f54779779-2ccqr\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:51.757466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:51.757429 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:52.076688 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:52.076596 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 21:49:52.079644 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:49:52.079615 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9e9abf_5904_4bb5_ac7c_6fff634c0a21.slice/crio-63d424bf29f7512914cc57c2920ce2c82c27690717ba219f74b1b378efb762df WatchSource:0}: Error finding container 63d424bf29f7512914cc57c2920ce2c82c27690717ba219f74b1b378efb762df: Status 404 returned error can't find the container with id 63d424bf29f7512914cc57c2920ce2c82c27690717ba219f74b1b378efb762df Oct 29 21:49:52.192325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:52.192294 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" event={"ID":"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21","Type":"ContainerStarted","Data":"63d424bf29f7512914cc57c2920ce2c82c27690717ba219f74b1b378efb762df"} Oct 29 21:49:54.525483 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:54.525443 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:49:54.525781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:54.525516 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:49:54.525781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:54.525555 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:49:55.189729 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.189696 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:49:55.207739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.207706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" event={"ID":"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21","Type":"ContainerStarted","Data":"4f2d1d058323b4c2dc6100a5dfd82621c3a4c3c3a603d7d7c03553003829893c"} Oct 29 21:49:55.299977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.299884 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podStartSLOduration=1.857106178 podStartE2EDuration="4.299864151s" totalImagesPullingTime="2.442757973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:55.299378722 +0000 UTC m=+454.147855572" watchObservedRunningTime="2026-10-29 21:49:55.299864151 +0000 UTC m=+454.148340997" Oct 29 21:49:55.758201 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.758105 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:49:55.760028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.759986 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-2ccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:49:55.760148 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:55.760062 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 29 21:49:56.180886 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.180806 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m"] Oct 29 21:49:56.184166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.184146 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.219318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.219282 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m"] Oct 29 21:49:56.289336 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.289297 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jxh\" (UniqueName: \"kubernetes.io/projected/3ad22e41-a5e2-402f-b9db-4b9138de4b58-kube-api-access-m5jxh\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.289530 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.289420 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad22e41-a5e2-402f-b9db-4b9138de4b58-openshift-service-ca\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.289599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.289525 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-maas-api-tls\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.289599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.289565 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-metrics-certs\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.390251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.390213 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad22e41-a5e2-402f-b9db-4b9138de4b58-openshift-service-ca\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.390475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.390289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-maas-api-tls\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.390475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.390324 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-metrics-certs\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.390475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.390381 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jxh\" (UniqueName: \"kubernetes.io/projected/3ad22e41-a5e2-402f-b9db-4b9138de4b58-kube-api-access-m5jxh\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.391050 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.391028 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad22e41-a5e2-402f-b9db-4b9138de4b58-openshift-service-ca\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.392803 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.392779 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-metrics-certs\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.392888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.392830 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad22e41-a5e2-402f-b9db-4b9138de4b58-maas-api-tls\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.416214 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.416174 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jxh\" (UniqueName: \"kubernetes.io/projected/3ad22e41-a5e2-402f-b9db-4b9138de4b58-kube-api-access-m5jxh\") pod \"maas-api-85cfbb945d-c2b2m\" (UID: \"3ad22e41-a5e2-402f-b9db-4b9138de4b58\") " pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.497748 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.497700 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:56.758717 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.758623 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-2ccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:49:56.758717 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.758684 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 29 21:49:56.826426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:56.826392 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m"] Oct 29 21:49:56.829361 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:49:56.829331 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad22e41_a5e2_402f_b9db_4b9138de4b58.slice/crio-485201fbf2ff7967ac473eaace0c270b1cc673ab705fcddc867ffe4e1b66db35 WatchSource:0}: Error finding container 485201fbf2ff7967ac473eaace0c270b1cc673ab705fcddc867ffe4e1b66db35: Status 404 returned error can't find the container with id 485201fbf2ff7967ac473eaace0c270b1cc673ab705fcddc867ffe4e1b66db35 Oct 29 21:49:57.216461 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:57.216425 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" event={"ID":"3ad22e41-a5e2-402f-b9db-4b9138de4b58","Type":"ContainerStarted","Data":"485201fbf2ff7967ac473eaace0c270b1cc673ab705fcddc867ffe4e1b66db35"} Oct 29 21:49:57.758493 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:57.758446 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-2ccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:49:57.758682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:57.758521 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 29 21:49:58.222774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:58.222736 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" event={"ID":"3ad22e41-a5e2-402f-b9db-4b9138de4b58","Type":"ContainerStarted","Data":"e707b550894bcef20c3460f440c389ce9d73b11b80682667edb82fdd2f051c63"} Oct 29 21:49:58.223174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:58.222830 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:49:58.285988 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:58.285933 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" podStartSLOduration=1.837945312 podStartE2EDuration="2.285914624s" totalImagesPullingTime="447.969312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:58.284221152 +0000 UTC m=+457.132697997" watchObservedRunningTime="2026-10-29 21:49:58.285914624 +0000 UTC m=+457.134391460" Oct 29 21:49:58.758431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:58.758380 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-2ccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:49:58.758661 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:58.758470 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 29 21:49:59.758328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:59.758292 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-2ccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:49:59.758739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:49:59.758349 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 29 21:50:00.762650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:00.762620 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:50:00.763028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:00.762894 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:50:00.763538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:00.763518 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 21:50:04.233804 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.233775 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cfbb945d-c2b2m" Oct 29 21:50:04.317341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.317304 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:50:04.317610 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.317563 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" podUID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" containerName="maas-api" containerID="cri-o://acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795" gracePeriod=30 Oct 29 21:50:04.569425 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.569397 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:50:04.667489 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.667450 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\") pod \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " Oct 29 21:50:04.667489 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.667496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\") pod \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " Oct 29 21:50:04.667754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.667537 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\") pod \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " Oct 29 21:50:04.667754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.667570 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\" (UniqueName: \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\") pod \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\" (UID: \"9faafa00-c5c0-4619-ba68-7a1a4a71893c\") " Oct 29 21:50:04.667945 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.667891 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca" pod "9faafa00-c5c0-4619-ba68-7a1a4a71893c" (UID: "9faafa00-c5c0-4619-ba68-7a1a4a71893c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:50:04.669756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.669730 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls" pod "9faafa00-c5c0-4619-ba68-7a1a4a71893c" (UID: "9faafa00-c5c0-4619-ba68-7a1a4a71893c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:50:04.669845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.669752 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs" pod "9faafa00-c5c0-4619-ba68-7a1a4a71893c" (UID: "9faafa00-c5c0-4619-ba68-7a1a4a71893c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:50:04.669845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.669829 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2" pod "9faafa00-c5c0-4619-ba68-7a1a4a71893c" (UID: "9faafa00-c5c0-4619-ba68-7a1a4a71893c"). InnerVolumeSpecName "kube-api-access-fktr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:50:04.768700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.768600 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:50:04.768700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.768639 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktr2\" (UniqueName: \"kubernetes.io/projected/9faafa00-c5c0-4619-ba68-7a1a4a71893c-kube-api-access-fktr2\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:50:04.768700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.768650 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9faafa00-c5c0-4619-ba68-7a1a4a71893c-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:50:04.768700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:04.768659 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faafa00-c5c0-4619-ba68-7a1a4a71893c-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:50:05.254429 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.254394 4376 generic.go:356] "Generic (PLEG): container finished" podID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" containerID="acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795" exitCode=0 Oct 29 21:50:05.254934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.254443 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" Oct 29 21:50:05.254934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.254480 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" event={"ID":"9faafa00-c5c0-4619-ba68-7a1a4a71893c","Type":"ContainerDied","Data":"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795"} Oct 29 21:50:05.254934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.254577 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj" event={"ID":"9faafa00-c5c0-4619-ba68-7a1a4a71893c","Type":"ContainerDied","Data":"de2b47b3a77d3066032bd1952a56ea640b0985c80bc1c1fbbb35486f4ae517c6"} Oct 29 21:50:05.254934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.254614 4376 scope.go:122] "RemoveContainer" containerID="acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795" Oct 29 21:50:05.264025 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.264005 4376 scope.go:122] "RemoveContainer" containerID="acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795" Oct 29 21:50:05.264344 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:50:05.264324 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795\": container with ID starting with acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795 not found: ID does not exist" containerID="acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795" Oct 29 21:50:05.264405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.264358 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795"} err="failed to get container status \"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795\": rpc error: code = NotFound desc = could not find container \"acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795\": container with ID starting with acb1a3de2ada8259ad7f7ac0bff1dba1c1c86ed6242948db1dab8e2875214795 not found: ID does not exist" Oct 29 21:50:05.313596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.313558 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:50:05.326137 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:05.326105 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b45cb84d-5g8bj"] Oct 29 21:50:07.270219 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:50:07.270188 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" path="/var/lib/kubelet/pods/9faafa00-c5c0-4619-ba68-7a1a4a71893c/volumes" Oct 29 21:50:16.434739 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:50:16.434696 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:50:46.432644 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:50:46.432541 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:51:13.562403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.562368 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:13.562819 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.562739 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" containerName="maas-api" Oct 29 21:51:13.562819 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.562751 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" containerName="maas-api" Oct 29 21:51:13.562819 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.562807 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9faafa00-c5c0-4619-ba68-7a1a4a71893c" containerName="maas-api" Oct 29 21:51:13.565721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.565704 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.585789 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.585761 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:13.596683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.596655 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 29 21:51:13.596838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.596658 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-cfzgs\"" Oct 29 21:51:13.596838 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.596771 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 29 21:51:13.665976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.665939 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.666143 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.666079 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84q2\" (UniqueName: \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.766753 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.766715 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84q2\" (UniqueName: \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.766975 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.766795 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.769312 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.769284 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.792086 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.792056 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84q2\" (UniqueName: \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\") pod \"ai-gateway-controller-7d85675f98-4phpm\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:13.878510 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:13.878427 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:14.176326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:14.176301 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:14.178985 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:51:14.178954 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae72a090_4103_4353_9af9_5b6051cf5c25.slice/crio-738c8d2e9735f7f26851e591229d0bd341596fb428c143988bad95f213b93b76 WatchSource:0}: Error finding container 738c8d2e9735f7f26851e591229d0bd341596fb428c143988bad95f213b93b76: Status 404 returned error can't find the container with id 738c8d2e9735f7f26851e591229d0bd341596fb428c143988bad95f213b93b76 Oct 29 21:51:14.557092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:14.557052 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" event={"ID":"ae72a090-4103-4353-9af9-5b6051cf5c25","Type":"ContainerStarted","Data":"738c8d2e9735f7f26851e591229d0bd341596fb428c143988bad95f213b93b76"} Oct 29 21:51:16.432882 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:51:16.432847 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:51:16.568025 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:16.567985 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" event={"ID":"ae72a090-4103-4353-9af9-5b6051cf5c25","Type":"ContainerStarted","Data":"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400"} Oct 29 21:51:16.568207 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:16.568099 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:16.610454 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:16.610401 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" podStartSLOduration=2.073716292 podStartE2EDuration="3.610387841s" totalImagesPullingTime="1.536671549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:16.609310504 +0000 UTC m=+535.457787348" watchObservedRunningTime="2026-10-29 21:51:16.610387841 +0000 UTC m=+535.458864685" Oct 29 21:51:18.137208 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.137169 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm"] Oct 29 21:51:18.141198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.141162 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.170144 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.170122 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 29 21:51:18.170144 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.170124 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lv89x\"" Oct 29 21:51:18.171871 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.171851 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm"] Oct 29 21:51:18.231692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.231662 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6cb7b"] Oct 29 21:51:18.235328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.235306 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.257672 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.257641 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6cb7b"] Oct 29 21:51:18.310734 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.310694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f77ed1-72e6-425b-9967-ef791198a32b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.310931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.310850 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdlg\" (UniqueName: \"kubernetes.io/projected/b0f77ed1-72e6-425b-9967-ef791198a32b-kube-api-access-kvdlg\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.411678 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.411595 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f77ed1-72e6-425b-9967-ef791198a32b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.411854 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.411793 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdlg\" (UniqueName: \"kubernetes.io/projected/b0f77ed1-72e6-425b-9967-ef791198a32b-kube-api-access-kvdlg\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.411854 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.411851 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89m8\" (UniqueName: \"kubernetes.io/projected/aaa82eab-67c3-48b0-9141-dacf6d9d4262-kube-api-access-w89m8\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.412018 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.411959 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa82eab-67c3-48b0-9141-dacf6d9d4262-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.412346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.412329 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f77ed1-72e6-425b-9967-ef791198a32b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.437213 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.437188 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdlg\" (UniqueName: \"kubernetes.io/projected/b0f77ed1-72e6-425b-9967-ef791198a32b-kube-api-access-kvdlg\") pod \"payload-pre-processing-7b98bb678c-9w5vm\" (UID: \"b0f77ed1-72e6-425b-9967-ef791198a32b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.456032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.456008 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:18.512816 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.512776 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89m8\" (UniqueName: \"kubernetes.io/projected/aaa82eab-67c3-48b0-9141-dacf6d9d4262-kube-api-access-w89m8\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.512971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.512838 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa82eab-67c3-48b0-9141-dacf6d9d4262-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.514047 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.514006 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa82eab-67c3-48b0-9141-dacf6d9d4262-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.538700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.538668 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89m8\" (UniqueName: \"kubernetes.io/projected/aaa82eab-67c3-48b0-9141-dacf6d9d4262-kube-api-access-w89m8\") pod \"payload-processing-5f78687d5b-6cb7b\" (UID: \"aaa82eab-67c3-48b0-9141-dacf6d9d4262\") " pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.548260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.548233 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:18.753452 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.753417 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm"] Oct 29 21:51:18.756754 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:51:18.756726 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f77ed1_72e6_425b_9967_ef791198a32b.slice/crio-2006e53cff969e51015d83b7fde16cef263e85bf64bbf65824c6a9b17f40fc71 WatchSource:0}: Error finding container 2006e53cff969e51015d83b7fde16cef263e85bf64bbf65824c6a9b17f40fc71: Status 404 returned error can't find the container with id 2006e53cff969e51015d83b7fde16cef263e85bf64bbf65824c6a9b17f40fc71 Oct 29 21:51:18.846981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:18.846956 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6cb7b"] Oct 29 21:51:18.849522 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:51:18.849497 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa82eab_67c3_48b0_9141_dacf6d9d4262.slice/crio-245aa0696708c80e2b5354a4763aa0e0357fe8c51fcb1849685553d69fdee4d4 WatchSource:0}: Error finding container 245aa0696708c80e2b5354a4763aa0e0357fe8c51fcb1849685553d69fdee4d4: Status 404 returned error can't find the container with id 245aa0696708c80e2b5354a4763aa0e0357fe8c51fcb1849685553d69fdee4d4 Oct 29 21:51:19.583215 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:19.583182 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" event={"ID":"b0f77ed1-72e6-425b-9967-ef791198a32b","Type":"ContainerStarted","Data":"2006e53cff969e51015d83b7fde16cef263e85bf64bbf65824c6a9b17f40fc71"} Oct 29 21:51:19.584850 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:19.584814 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" event={"ID":"aaa82eab-67c3-48b0-9141-dacf6d9d4262","Type":"ContainerStarted","Data":"245aa0696708c80e2b5354a4763aa0e0357fe8c51fcb1849685553d69fdee4d4"} Oct 29 21:51:21.596319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.596281 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" event={"ID":"aaa82eab-67c3-48b0-9141-dacf6d9d4262","Type":"ContainerStarted","Data":"7f8560ef86cef6db617c211b32b623982850485c4a853f55f2c2ad1dd0a76dd1"} Oct 29 21:51:21.596764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.596383 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:21.597761 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.597742 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" event={"ID":"b0f77ed1-72e6-425b-9967-ef791198a32b","Type":"ContainerStarted","Data":"f30936e881d73f4b40bdd849050564f356e6c4dea5021d169c0781ad26a2ae5f"} Oct 29 21:51:21.597894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.597876 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:21.647740 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.647672 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" podStartSLOduration=1.479509937 podStartE2EDuration="3.647656022s" totalImagesPullingTime="2.168146085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:21.646429573 +0000 UTC m=+540.494906418" watchObservedRunningTime="2026-10-29 21:51:21.647656022 +0000 UTC m=+540.496132865" Oct 29 21:51:21.689103 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:21.689050 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" podStartSLOduration=1.394226323 podStartE2EDuration="3.689035012s" totalImagesPullingTime="2.294808689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:21.688487156 +0000 UTC m=+540.536964001" watchObservedRunningTime="2026-10-29 21:51:21.689035012 +0000 UTC m=+540.537511856" Oct 29 21:51:27.574446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.574411 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:27.882521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.882441 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-54b58f7c85-hgxhj"] Oct 29 21:51:27.887383 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.887349 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:27.895862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.895839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfc987-b018-4040-bdb2-8dc46d06dedb-metrics-certs\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:27.895953 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.895916 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfks\" (UniqueName: \"kubernetes.io/projected/f9cfc987-b018-4040-bdb2-8dc46d06dedb-kube-api-access-ztfks\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:27.918280 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.918245 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54b58f7c85-hgxhj"] Oct 29 21:51:27.997055 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.997023 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfks\" (UniqueName: \"kubernetes.io/projected/f9cfc987-b018-4040-bdb2-8dc46d06dedb-kube-api-access-ztfks\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:27.997240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.997113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfc987-b018-4040-bdb2-8dc46d06dedb-metrics-certs\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:27.999539 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:27.999515 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfc987-b018-4040-bdb2-8dc46d06dedb-metrics-certs\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:28.027343 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:28.027315 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfks\" (UniqueName: \"kubernetes.io/projected/f9cfc987-b018-4040-bdb2-8dc46d06dedb-kube-api-access-ztfks\") pod \"ai-gateway-controller-54b58f7c85-hgxhj\" (UID: \"f9cfc987-b018-4040-bdb2-8dc46d06dedb\") " pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:28.201448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:28.201421 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:28.501686 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:28.501657 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54b58f7c85-hgxhj"] Oct 29 21:51:28.504387 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:51:28.504359 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cfc987_b018_4040_bdb2_8dc46d06dedb.slice/crio-c211d0825f38c7e9a43e9665ba2a2506b322cf58b92df710468e52540e0f4502 WatchSource:0}: Error finding container c211d0825f38c7e9a43e9665ba2a2506b322cf58b92df710468e52540e0f4502: Status 404 returned error can't find the container with id c211d0825f38c7e9a43e9665ba2a2506b322cf58b92df710468e52540e0f4502 Oct 29 21:51:28.628137 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:28.628103 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" event={"ID":"f9cfc987-b018-4040-bdb2-8dc46d06dedb","Type":"ContainerStarted","Data":"c211d0825f38c7e9a43e9665ba2a2506b322cf58b92df710468e52540e0f4502"} Oct 29 21:51:29.633314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:29.633279 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" event={"ID":"f9cfc987-b018-4040-bdb2-8dc46d06dedb","Type":"ContainerStarted","Data":"efbfc4329a87e99060625239aed01d23104e1d2ba816f2ea2c8981c44819c22f"} Oct 29 21:51:29.633794 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:29.633417 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:29.676727 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:29.676678 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" podStartSLOduration=2.233159632 podStartE2EDuration="2.676665044s" totalImagesPullingTime="443.505412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:29.675707504 +0000 UTC m=+548.524184349" watchObservedRunningTime="2026-10-29 21:51:29.676665044 +0000 UTC m=+548.525141888" Oct 29 21:51:32.605754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:32.605728 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w5vm" Oct 29 21:51:32.606151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:32.605790 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-6cb7b" Oct 29 21:51:40.641158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:40.641125 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-54b58f7c85-hgxhj" Oct 29 21:51:40.713524 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:40.713494 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:40.713691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:40.713671 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" podUID="ae72a090-4103-4353-9af9-5b6051cf5c25" containerName="manager" containerID="cri-o://9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400" gracePeriod=30 Oct 29 21:51:40.953524 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:40.953494 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:41.016185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.016139 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\" (UniqueName: \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\") pod \"ae72a090-4103-4353-9af9-5b6051cf5c25\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " Oct 29 21:51:41.016375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.016325 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\") pod \"ae72a090-4103-4353-9af9-5b6051cf5c25\" (UID: \"ae72a090-4103-4353-9af9-5b6051cf5c25\") " Oct 29 21:51:41.018346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.018322 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2" pod "ae72a090-4103-4353-9af9-5b6051cf5c25" (UID: "ae72a090-4103-4353-9af9-5b6051cf5c25"). InnerVolumeSpecName "kube-api-access-b84q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:51:41.018417 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.018337 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs" pod "ae72a090-4103-4353-9af9-5b6051cf5c25" (UID: "ae72a090-4103-4353-9af9-5b6051cf5c25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:51:41.117610 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.117576 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84q2\" (UniqueName: \"kubernetes.io/projected/ae72a090-4103-4353-9af9-5b6051cf5c25-kube-api-access-b84q2\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:51:41.117610 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.117606 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae72a090-4103-4353-9af9-5b6051cf5c25-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:51:41.688406 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.688312 4376 generic.go:356] "Generic (PLEG): container finished" podID="ae72a090-4103-4353-9af9-5b6051cf5c25" containerID="9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400" exitCode=0 Oct 29 21:51:41.688406 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.688391 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" Oct 29 21:51:41.688940 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.688392 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" event={"ID":"ae72a090-4103-4353-9af9-5b6051cf5c25","Type":"ContainerDied","Data":"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400"} Oct 29 21:51:41.688940 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.688511 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4phpm" event={"ID":"ae72a090-4103-4353-9af9-5b6051cf5c25","Type":"ContainerDied","Data":"738c8d2e9735f7f26851e591229d0bd341596fb428c143988bad95f213b93b76"} Oct 29 21:51:41.688940 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.688531 4376 scope.go:122] "RemoveContainer" containerID="9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400" Oct 29 21:51:41.696542 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.696525 4376 scope.go:122] "RemoveContainer" containerID="9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400" Oct 29 21:51:41.696817 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:51:41.696797 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400\": container with ID starting with 9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400 not found: ID does not exist" containerID="9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400" Oct 29 21:51:41.696887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.696825 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400"} err="failed to get container status \"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400\": rpc error: code = NotFound desc = could not find container \"9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400\": container with ID starting with 9bedcbce780aeed4db4ae04735544792da0f0ae5c075c7a859146db5dca49400 not found: ID does not exist" Oct 29 21:51:41.743517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.743486 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:41.755886 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:41.755857 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4phpm"] Oct 29 21:51:43.270063 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:43.270030 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae72a090-4103-4353-9af9-5b6051cf5c25" path="/var/lib/kubelet/pods/ae72a090-4103-4353-9af9-5b6051cf5c25/volumes" Oct 29 21:51:46.431248 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:51:46.431208 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:51:58.971713 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.971663 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4"] Oct 29 21:51:58.975815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.972195 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ae72a090-4103-4353-9af9-5b6051cf5c25" containerName="manager" Oct 29 21:51:58.975815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.972216 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae72a090-4103-4353-9af9-5b6051cf5c25" containerName="manager" Oct 29 21:51:58.975815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.972329 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae72a090-4103-4353-9af9-5b6051cf5c25" containerName="manager" Oct 29 21:51:58.976065 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.975831 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:58.992636 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:58.992601 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 21:51:59.004224 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.004195 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwg6\"" Oct 29 21:51:59.004224 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.004195 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 21:51:59.004505 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.004491 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 29 21:51:59.006549 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.006527 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4"] Oct 29 21:51:59.068061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068022 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.068259 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068074 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.068259 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068159 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.068259 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068225 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837cb92-b1eb-4228-99d1-80b215749ef2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.068408 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068276 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.068408 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.068302 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8whd\" (UniqueName: \"kubernetes.io/projected/c837cb92-b1eb-4228-99d1-80b215749ef2-kube-api-access-t8whd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169454 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169418 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169464 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8whd\" (UniqueName: \"kubernetes.io/projected/c837cb92-b1eb-4228-99d1-80b215749ef2-kube-api-access-t8whd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169514 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169554 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169602 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.169637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169633 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837cb92-b1eb-4228-99d1-80b215749ef2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.170019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.169994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.170019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.170009 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.170138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.170030 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.171738 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.171714 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837cb92-b1eb-4228-99d1-80b215749ef2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.172006 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.171988 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837cb92-b1eb-4228-99d1-80b215749ef2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.195227 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.195206 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8whd\" (UniqueName: \"kubernetes.io/projected/c837cb92-b1eb-4228-99d1-80b215749ef2-kube-api-access-t8whd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4\" (UID: \"c837cb92-b1eb-4228-99d1-80b215749ef2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.289587 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.289498 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:51:59.591885 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.591808 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4"] Oct 29 21:51:59.594838 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:51:59.594815 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc837cb92_b1eb_4228_99d1_80b215749ef2.slice/crio-f561137ce256cdadfff6b7fc93678c67dd78889f486b505875925efa387e5649 WatchSource:0}: Error finding container f561137ce256cdadfff6b7fc93678c67dd78889f486b505875925efa387e5649: Status 404 returned error can't find the container with id f561137ce256cdadfff6b7fc93678c67dd78889f486b505875925efa387e5649 Oct 29 21:51:59.769258 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:51:59.769221 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" event={"ID":"c837cb92-b1eb-4228-99d1-80b215749ef2","Type":"ContainerStarted","Data":"f561137ce256cdadfff6b7fc93678c67dd78889f486b505875925efa387e5649"} Oct 29 21:52:01.780991 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:01.780949 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" event={"ID":"c837cb92-b1eb-4228-99d1-80b215749ef2","Type":"ContainerStarted","Data":"977a5dd9c95325aaf5a196161caf59b4c2e7523a0728da664aaa2dc027d0d359"} Oct 29 21:52:01.825724 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:01.825676 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" podStartSLOduration=2.608428622 podStartE2EDuration="3.825663078s" totalImagesPullingTime="1.217234456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:01.824248251 +0000 UTC m=+580.672725119" watchObservedRunningTime="2026-10-29 21:52:01.825663078 +0000 UTC m=+580.674139922" Oct 29 21:52:02.290579 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:02.290548 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:52:05.971926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:05.971733 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt"] Oct 29 21:52:05.975976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:05.975953 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:05.992626 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:05.992604 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 29 21:52:06.006194 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.006160 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt"] Oct 29 21:52:06.150486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150451 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfwb\" (UniqueName: \"kubernetes.io/projected/06bc08b1-272a-45fd-981f-69fe88948bc0-kube-api-access-mlfwb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.150665 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150504 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.150736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150671 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.150736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150724 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.150899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150879 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bc08b1-272a-45fd-981f-69fe88948bc0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.150982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.150945 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251712 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251628 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfwb\" (UniqueName: \"kubernetes.io/projected/06bc08b1-272a-45fd-981f-69fe88948bc0-kube-api-access-mlfwb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251712 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251683 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251959 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251726 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251959 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251757 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251959 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251835 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bc08b1-272a-45fd-981f-69fe88948bc0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.251959 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.251860 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.252197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.252170 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.252261 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.252198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.252308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.252267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.254232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.254200 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bc08b1-272a-45fd-981f-69fe88948bc0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.254232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.254215 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bc08b1-272a-45fd-981f-69fe88948bc0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.277218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.277191 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfwb\" (UniqueName: \"kubernetes.io/projected/06bc08b1-272a-45fd-981f-69fe88948bc0-kube-api-access-mlfwb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q8tnt\" (UID: \"06bc08b1-272a-45fd-981f-69fe88948bc0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.288899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.288876 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:06.590381 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.590353 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt"] Oct 29 21:52:06.592886 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:06.592854 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bc08b1_272a_45fd_981f_69fe88948bc0.slice/crio-eaf84b1c646d16e0a5ebe65475dc90eb503b011a5e93ce73ec679e6d97bcdcb6 WatchSource:0}: Error finding container eaf84b1c646d16e0a5ebe65475dc90eb503b011a5e93ce73ec679e6d97bcdcb6: Status 404 returned error can't find the container with id eaf84b1c646d16e0a5ebe65475dc90eb503b011a5e93ce73ec679e6d97bcdcb6 Oct 29 21:52:06.803664 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:06.803627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" event={"ID":"06bc08b1-272a-45fd-981f-69fe88948bc0","Type":"ContainerStarted","Data":"eaf84b1c646d16e0a5ebe65475dc90eb503b011a5e93ce73ec679e6d97bcdcb6"} Oct 29 21:52:07.810053 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:07.810007 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" event={"ID":"06bc08b1-272a-45fd-981f-69fe88948bc0","Type":"ContainerStarted","Data":"5d79f32ebf7ad68b629d2f986f307c3f6a51351d380bdb0582976e941cfa529d"} Oct 29 21:52:07.855135 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:07.855082 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" podStartSLOduration=2.635545101 podStartE2EDuration="2.855066733s" totalImagesPullingTime="219.521632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:07.853612622 +0000 UTC m=+586.702089467" watchObservedRunningTime="2026-10-29 21:52:07.855066733 +0000 UTC m=+586.703543612" Oct 29 21:52:08.021226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.021192 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5"] Oct 29 21:52:08.034120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.034093 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.047380 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.047347 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5"] Oct 29 21:52:08.052553 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.052527 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 29 21:52:08.168966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.168842 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkln\" (UniqueName: \"kubernetes.io/projected/75921689-7442-45e6-b3ce-8b58c947ba78-kube-api-access-2fkln\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.168966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.168940 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.169174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.169000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75921689-7442-45e6-b3ce-8b58c947ba78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.169174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.169102 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.169174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.169131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.169174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.169159 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.269895 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.269843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkln\" (UniqueName: \"kubernetes.io/projected/75921689-7442-45e6-b3ce-8b58c947ba78-kube-api-access-2fkln\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.269937 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.269958 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75921689-7442-45e6-b3ce-8b58c947ba78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.269989 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.270010 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.270034 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270470 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.270439 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270531 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.270479 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.270531 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.270494 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.272357 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.272330 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75921689-7442-45e6-b3ce-8b58c947ba78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.272707 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.272686 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75921689-7442-45e6-b3ce-8b58c947ba78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.289174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.289144 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:08.296893 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.296864 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkln\" (UniqueName: \"kubernetes.io/projected/75921689-7442-45e6-b3ce-8b58c947ba78-kube-api-access-2fkln\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5\" (UID: \"75921689-7442-45e6-b3ce-8b58c947ba78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.350730 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.350694 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:08.652592 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.652569 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5"] Oct 29 21:52:08.654823 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:08.654797 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75921689_7442_45e6_b3ce_8b58c947ba78.slice/crio-956bd7b37a3c9b0d2d698326fab3c335b0ea186bd0185a6d05e4c17f7830da3b WatchSource:0}: Error finding container 956bd7b37a3c9b0d2d698326fab3c335b0ea186bd0185a6d05e4c17f7830da3b: Status 404 returned error can't find the container with id 956bd7b37a3c9b0d2d698326fab3c335b0ea186bd0185a6d05e4c17f7830da3b Oct 29 21:52:08.817977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:08.817938 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" event={"ID":"75921689-7442-45e6-b3ce-8b58c947ba78","Type":"ContainerStarted","Data":"956bd7b37a3c9b0d2d698326fab3c335b0ea186bd0185a6d05e4c17f7830da3b"} Oct 29 21:52:09.290525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:09.290483 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:52:09.306051 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:09.306022 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:52:09.824014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:09.823975 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" event={"ID":"75921689-7442-45e6-b3ce-8b58c947ba78","Type":"ContainerStarted","Data":"6fd50fcd6a3b8529809d65c39a01208d2ec42cf078af453e67a1180671151908"} Oct 29 21:52:09.835068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:09.835044 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4" Oct 29 21:52:09.869217 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:09.869163 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" podStartSLOduration=2.656825444 podStartE2EDuration="2.86914915s" totalImagesPullingTime="212.323706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:09.868183388 +0000 UTC m=+588.716660255" watchObservedRunningTime="2026-10-29 21:52:09.86914915 +0000 UTC m=+588.717625994" Oct 29 21:52:10.351383 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.351340 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:10.371657 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.371625 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xczl2"] Oct 29 21:52:10.375686 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.375656 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.391989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.391958 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8v9t\" (UniqueName: \"kubernetes.io/projected/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-kube-api-access-j8v9t\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.391989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.391993 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.392214 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.392020 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.392214 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.392126 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-home\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.392315 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.392257 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.392373 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.392315 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.396789 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.396747 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xczl2"] Oct 29 21:52:10.404192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.404168 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 29 21:52:10.493596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493563 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.493812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493613 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-home\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.493812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493672 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.493812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493710 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.493812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8v9t\" (UniqueName: \"kubernetes.io/projected/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-kube-api-access-j8v9t\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.493812 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.493800 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.494228 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.494205 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-home\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.494306 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.494253 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.494306 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.494261 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.496018 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.495990 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.496255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.496238 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.518826 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.518793 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8v9t\" (UniqueName: \"kubernetes.io/projected/1ae6e11a-dcdc-47bc-baaf-05f17813bdfa-kube-api-access-j8v9t\") pod \"echo-simulator-kserve-64b6f8f96c-xczl2\" (UID: \"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.691070 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.690966 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:10.994014 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:10.993981 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xczl2"] Oct 29 21:52:10.996964 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:10.996937 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae6e11a_dcdc_47bc_baaf_05f17813bdfa.slice/crio-0a9488e9ba8b5e2cd867d25e5b49d977ca9ffded56aca65b7456d82b1dfeecf4 WatchSource:0}: Error finding container 0a9488e9ba8b5e2cd867d25e5b49d977ca9ffded56aca65b7456d82b1dfeecf4: Status 404 returned error can't find the container with id 0a9488e9ba8b5e2cd867d25e5b49d977ca9ffded56aca65b7456d82b1dfeecf4 Oct 29 21:52:11.843922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:11.843883 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" event={"ID":"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa","Type":"ContainerStarted","Data":"0a9488e9ba8b5e2cd867d25e5b49d977ca9ffded56aca65b7456d82b1dfeecf4"} Oct 29 21:52:12.850791 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:12.850761 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" event={"ID":"1ae6e11a-dcdc-47bc-baaf-05f17813bdfa","Type":"ContainerStarted","Data":"842b4a1c310b5817fdd334103ee40ed6ca254097e8b2df1c668316832fe71aa1"} Oct 29 21:52:12.900956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:12.900886 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" podStartSLOduration=1.131903072 podStartE2EDuration="2.900872764s" totalImagesPullingTime="1.768969692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:12.899404906 +0000 UTC m=+591.747881763" watchObservedRunningTime="2026-10-29 21:52:12.900872764 +0000 UTC m=+591.749349633" Oct 29 21:52:13.691589 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:13.691558 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:16.290197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:16.290152 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:16.302736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:16.302708 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:16.432041 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:52:16.432006 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:52:16.887530 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:16.887493 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q8tnt" Oct 29 21:52:18.351036 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:18.351008 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:18.363032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:18.363007 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:18.890178 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:18.890141 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5" Oct 29 21:52:20.692173 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:20.692132 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:20.704863 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:20.704837 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:20.901402 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:20.901372 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xczl2" Oct 29 21:52:21.883025 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:21.882994 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx"] Oct 29 21:52:21.887843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:21.887822 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:21.910696 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:21.910624 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx"] Oct 29 21:52:22.010227 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010227 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010233 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010334 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz27p\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-kube-api-access-qz27p\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010394 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010434 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010493 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.010599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.010556 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff83b6-0644-419f-9452-ac45a35976ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111366 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111321 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff83b6-0644-419f-9452-ac45a35976ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111427 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111473 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111500 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz27p\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-kube-api-access-qz27p\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111527 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111556 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111602 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.111859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.111641 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.112061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.112022 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.112061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.112042 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.112175 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.112143 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff83b6-0644-419f-9452-ac45a35976ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.112175 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.112164 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.112263 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.112207 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.114260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.114230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.114451 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.114434 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.138326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.138254 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.138507 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.138484 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz27p\" (UniqueName: \"kubernetes.io/projected/f7ff83b6-0644-419f-9452-ac45a35976ee-kube-api-access-qz27p\") pod \"maas-default-gateway-openshift-default-6f54779779-kmrzx\" (UID: \"f7ff83b6-0644-419f-9452-ac45a35976ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.202995 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.202952 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:22.522310 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.522284 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx"] Oct 29 21:52:22.524602 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:22.524575 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff83b6_0644_419f_9452_ac45a35976ee.slice/crio-6f30ef955f79ea6510bdf134dfe2441f8d7ed2b32cf038440a033e14fcb81dcb WatchSource:0}: Error finding container 6f30ef955f79ea6510bdf134dfe2441f8d7ed2b32cf038440a033e14fcb81dcb: Status 404 returned error can't find the container with id 6f30ef955f79ea6510bdf134dfe2441f8d7ed2b32cf038440a033e14fcb81dcb Oct 29 21:52:22.526855 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.526821 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:22.526964 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.526922 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:22.527034 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.526968 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:22.901520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.901436 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" event={"ID":"f7ff83b6-0644-419f-9452-ac45a35976ee","Type":"ContainerStarted","Data":"b859badc519991592016f62a35c7d12be46d372a14766369c584c414fae4c522"} Oct 29 21:52:22.901520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.901487 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" event={"ID":"f7ff83b6-0644-419f-9452-ac45a35976ee","Type":"ContainerStarted","Data":"6f30ef955f79ea6510bdf134dfe2441f8d7ed2b32cf038440a033e14fcb81dcb"} Oct 29 21:52:22.948829 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:22.948773 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podStartSLOduration=1.948755656 podStartE2EDuration="1.948755656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:22.947284184 +0000 UTC m=+601.795761039" watchObservedRunningTime="2026-10-29 21:52:22.948755656 +0000 UTC m=+601.797232498" Oct 29 21:52:23.204068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:23.204040 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:23.205598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:23.205577 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:23.205692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:23.205617 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:24.204352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:24.204315 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:24.204728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:24.204393 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:25.203537 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:25.203499 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:25.203721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:25.203568 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:26.203803 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:26.203765 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:26.204216 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:26.203826 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:27.203650 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:27.203614 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:27.203830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:27.203671 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:28.203858 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:28.203822 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:28.204286 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:28.203883 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:29.203800 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:29.203762 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kmrzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 21:52:29.204005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:29.203828 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" podUID="f7ff83b6-0644-419f-9452-ac45a35976ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 21:52:30.561762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:30.561726 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:30.562251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:30.562054 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:30.562927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:30.562884 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kmrzx" Oct 29 21:52:36.953648 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:36.953613 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 21:52:36.957233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:36.957212 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:36.980645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:36.980614 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 21:52:37.079058 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079019 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079215 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079081 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079256 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079203 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079329 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079309 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079355 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079402 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079384 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079476 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079461 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079518 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079503 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.079604 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.079588 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jt47\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.180657 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jt47\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.180840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180689 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.180840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180737 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.180840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180763 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.180840 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180812 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.180985 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181029 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181269 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181456 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181364 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.181528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.181512 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.183185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.183163 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.183342 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.183323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.206219 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.206167 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.206346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.206329 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jt47\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\") pod \"maas-default-gateway-openshift-default-6f54779779-mkczb\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.273148 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.273120 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:37.592957 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.592925 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 21:52:37.595754 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:37.595718 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf087dc1_2f0e_4838_b84c_a7386e69815c.slice/crio-080500c8bf9165868dbec362f20537da6032d539679921d36338fee5cef327e4 WatchSource:0}: Error finding container 080500c8bf9165868dbec362f20537da6032d539679921d36338fee5cef327e4: Status 404 returned error can't find the container with id 080500c8bf9165868dbec362f20537da6032d539679921d36338fee5cef327e4 Oct 29 21:52:37.597948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.597898 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:37.598019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.597983 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:37.598019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.598012 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:37.975675 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.975631 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" event={"ID":"df087dc1-2f0e-4838-b84c-a7386e69815c","Type":"ContainerStarted","Data":"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba"} Oct 29 21:52:37.976148 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:37.975734 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" event={"ID":"df087dc1-2f0e-4838-b84c-a7386e69815c","Type":"ContainerStarted","Data":"080500c8bf9165868dbec362f20537da6032d539679921d36338fee5cef327e4"} Oct 29 21:52:38.022290 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:38.022237 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podStartSLOduration=2.022223266 podStartE2EDuration="2.022223266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:38.021159604 +0000 UTC m=+616.869636487" watchObservedRunningTime="2026-10-29 21:52:38.022223266 +0000 UTC m=+616.870700110" Oct 29 21:52:38.273397 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:38.273311 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:38.274898 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:38.274876 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:38.274979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:38.274954 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:39.273488 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:39.273453 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:39.273926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:39.273514 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:40.274346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:40.274312 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:40.274786 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:40.274363 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:41.273648 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:41.273614 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:41.273816 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:41.273663 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:42.274138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:42.274097 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:42.274533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:42.274150 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:43.274428 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:43.274396 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:43.274826 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:43.274445 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:44.273533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:44.273493 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 21:52:44.273716 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:44.273563 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 21:52:45.667855 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:45.667814 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:45.668377 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:45.668321 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:45.669024 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:45.669007 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 21:52:46.432790 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:52:46.432752 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:52:51.998337 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:51.998299 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 21:52:52.005145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.005120 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.038112 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.038052 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 21:52:52.052132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052086 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052307 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052151 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052307 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052205 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052307 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052269 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052409 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052341 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052409 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052399 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052443 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2fz\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.052529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.052497 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.153714 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153662 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.153714 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153714 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153788 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2fz\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153831 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153859 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.153972 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154345 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154211 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154345 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154332 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154451 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154362 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154451 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154398 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154563 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.154686 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.154665 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.156453 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.156426 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.156671 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.156653 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.178855 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.178821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.179051 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.179029 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2fz\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\") pod \"maas-default-gateway-openshift-default-6f54779779-j5t99\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.320297 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.320208 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:52.640607 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.640582 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 21:52:52.642669 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:52:52.642645 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a26720_f408_4ad2_a6a6_01317092e316.slice/crio-0c372ebe6268d51edc18c6c50471fa608087530d34b8606bc1198158d6e009a0 WatchSource:0}: Error finding container 0c372ebe6268d51edc18c6c50471fa608087530d34b8606bc1198158d6e009a0: Status 404 returned error can't find the container with id 0c372ebe6268d51edc18c6c50471fa608087530d34b8606bc1198158d6e009a0 Oct 29 21:52:52.644873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.644842 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:52.644966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.644920 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:52.644966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:52.644950 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:52:53.054064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.054028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" event={"ID":"95a26720-f408-4ad2-a6a6-01317092e316","Type":"ContainerStarted","Data":"1fb6b5a5eef6dae2f278eaf765e80aa9193abeeea08a2c3a41dac235c2bd862e"} Oct 29 21:52:53.054507 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.054082 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" event={"ID":"95a26720-f408-4ad2-a6a6-01317092e316","Type":"ContainerStarted","Data":"0c372ebe6268d51edc18c6c50471fa608087530d34b8606bc1198158d6e009a0"} Oct 29 21:52:53.104765 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.104698 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podStartSLOduration=2.104677437 podStartE2EDuration="2.104677437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:53.100560741 +0000 UTC m=+631.949037588" watchObservedRunningTime="2026-10-29 21:52:53.104677437 +0000 UTC m=+631.953154285" Oct 29 21:52:53.321172 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.321075 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:52:53.322525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.322492 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:53.322651 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:53.322552 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:54.321396 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:54.321358 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:54.321771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:54.321422 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:55.321486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:55.321439 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:55.322043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:55.321507 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:56.321248 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:56.321213 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:56.321432 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:56.321267 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:57.321637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:57.321598 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:57.322040 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:57.321672 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:58.320660 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:58.320613 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:58.320859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:58.320678 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:52:59.321255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:59.321218 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 21:52:59.321688 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:52:59.321285 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 21:53:00.978565 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:00.978529 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:53:00.979215 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:00.979189 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:53:00.980204 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:00.980185 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 21:53:05.814371 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.814338 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:53:05.819507 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.819486 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.836773 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.836748 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2976cd0d-openshift-default-dockercfg-7psn2\"" Oct 29 21:53:05.867829 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.867797 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:53:05.898857 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.898828 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899049 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.898879 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899122 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899039 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899122 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899079 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899122 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899103 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899235 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899201 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899235 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899231 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr85l\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899298 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899266 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.899346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.899330 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:05.967067 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.967031 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:05.971338 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.971314 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:05.993990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.993962 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:05.999881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.999854 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.999926 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.999951 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:05.999986 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000020 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000247 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000223 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000333 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000265 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000330 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000372 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr85l\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000486 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000776 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000751 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000878 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000796 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000878 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.000971 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.000944 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.002672 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.002639 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.002814 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.002689 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.012574 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.012533 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-044e79d4-openshift-default-dockercfg-wjbk7\"" Oct 29 21:53:06.037220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.037195 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.037350 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.037286 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr85l\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\") pod \"e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.100987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.100889 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.100987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.100948 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101182 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101182 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101115 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101220 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101337 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101337 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101327 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101417 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101357 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsg4w\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.101467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.101444 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.134617 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.134590 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:06.202916 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.202870 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203069 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.202962 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203069 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203012 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203069 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203057 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203087 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203133 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsg4w\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203194 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203345 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203345 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203272 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203442 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203371 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.203663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.203638 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.204218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.204193 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.205135 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.205111 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.206316 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.206284 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.207529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.207507 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.208621 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.208593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.228635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.228607 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.229043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.229027 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsg4w\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\") pod \"e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.287071 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.285429 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:06.464234 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.464210 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:53:06.466980 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:06.466938 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b86e2b_8441_4ec1_b19b_b29b06a5f96a.slice/crio-05b212322d9ccee18ab2543787b71570b934f8dcb8d6eca0d2bd7abe9d5f0cb2 WatchSource:0}: Error finding container 05b212322d9ccee18ab2543787b71570b934f8dcb8d6eca0d2bd7abe9d5f0cb2: Status 404 returned error can't find the container with id 05b212322d9ccee18ab2543787b71570b934f8dcb8d6eca0d2bd7abe9d5f0cb2 Oct 29 21:53:06.469221 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.469185 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:06.469308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.469257 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:06.469308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.469291 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:06.638797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.638762 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:06.641596 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:06.641568 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70f06ff_f5fc_45bf_bfe8_c672404e97da.slice/crio-38fe489251ca633d4da4fb0109290ab8196a1bee44a67c91d3ff829d8b95d099 WatchSource:0}: Error finding container 38fe489251ca633d4da4fb0109290ab8196a1bee44a67c91d3ff829d8b95d099: Status 404 returned error can't find the container with id 38fe489251ca633d4da4fb0109290ab8196a1bee44a67c91d3ff829d8b95d099 Oct 29 21:53:06.644310 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.644265 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:06.644397 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.644365 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:06.644445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:06.644405 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:07.130715 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.130679 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerStarted","Data":"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b"} Oct 29 21:53:07.131207 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.130737 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerStarted","Data":"38fe489251ca633d4da4fb0109290ab8196a1bee44a67c91d3ff829d8b95d099"} Oct 29 21:53:07.132462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.132435 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerStarted","Data":"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf"} Oct 29 21:53:07.132578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.132486 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerStarted","Data":"05b212322d9ccee18ab2543787b71570b934f8dcb8d6eca0d2bd7abe9d5f0cb2"} Oct 29 21:53:07.134770 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.134753 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:07.139625 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.139608 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:07.182967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.182922 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podStartSLOduration=2.182895859 podStartE2EDuration="2.182895859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:07.181588069 +0000 UTC m=+646.030064930" watchObservedRunningTime="2026-10-29 21:53:07.182895859 +0000 UTC m=+646.031372705" Oct 29 21:53:07.233739 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.233692 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podStartSLOduration=2.233679153 podStartE2EDuration="2.233679153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:07.231989659 +0000 UTC m=+646.080466505" watchObservedRunningTime="2026-10-29 21:53:07.233679153 +0000 UTC m=+646.082156000" Oct 29 21:53:07.286301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.286275 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:07.291475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.291446 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:07.741182 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.741148 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 21:53:07.745094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.745073 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.771385 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.771359 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 21:53:07.790390 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.790369 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3b067-openshift-default-dockercfg-t4jcs\"" Oct 29 21:53:07.820426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820398 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.820587 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820435 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.820587 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820456 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.820587 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820560 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.821091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820660 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.821091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btjq\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.821091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820711 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.821091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820776 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.821091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.820839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921709 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921674 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921733 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921752 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921771 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921788 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.921873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921865 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922208 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921932 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btjq\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922208 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.921965 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922208 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922001 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922368 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922335 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922368 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922348 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922476 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922390 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922555 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.922671 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.922654 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.924308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.924285 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.924411 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.924291 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.946360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.946330 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:07.946474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:07.946459 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btjq\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:08.061717 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.061362 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:08.145302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.144950 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:08.145302 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.144994 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:08.145993 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.145943 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:08.146079 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.146012 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:08.413885 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.413862 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 21:53:08.415785 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:08.415753 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cdab92_ac02_434a_97fd_d6df904b2365.slice/crio-e4657668d7d6b5f21b6bf77fce9c7aad52b6d87169989e635905ac0d66c1bad3 WatchSource:0}: Error finding container e4657668d7d6b5f21b6bf77fce9c7aad52b6d87169989e635905ac0d66c1bad3: Status 404 returned error can't find the container with id e4657668d7d6b5f21b6bf77fce9c7aad52b6d87169989e635905ac0d66c1bad3 Oct 29 21:53:08.418260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.418220 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:08.418340 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.418309 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:08.418378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:08.418365 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:09.151089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:09.151039 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerStarted","Data":"d1ecc8e5a9251cdc26519b74a76428c5d0bbe202185efe2c5830e282b782d269"} Oct 29 21:53:09.151520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:09.151112 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerStarted","Data":"e4657668d7d6b5f21b6bf77fce9c7aad52b6d87169989e635905ac0d66c1bad3"} Oct 29 21:53:09.204566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:09.204495 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podStartSLOduration=2.20448093 podStartE2EDuration="2.20448093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:09.203518462 +0000 UTC m=+648.051995308" watchObservedRunningTime="2026-10-29 21:53:09.20448093 +0000 UTC m=+648.052957904" Oct 29 21:53:10.062332 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.062300 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:10.067360 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.067336 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:10.156451 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.156420 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:10.157185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.157166 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:10.664979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.664923 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:10.668985 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.668969 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.689241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.689217 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-229ab2ac\"" Oct 29 21:53:10.689562 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.689521 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-229ab2ac-dockercfg-98kf8\"" Oct 29 21:53:10.694471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.694438 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:10.751421 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.751392 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.751581 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.751429 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgs4\" (UniqueName: \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.814462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.814421 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:10.825192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.825152 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.832547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.832301 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:10.852523 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.852485 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4v6\" (UniqueName: \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.852675 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.852600 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.852675 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.852646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgs4\" (UniqueName: \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.852760 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.852703 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.853284 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.853266 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.877737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.877703 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgs4\" (UniqueName: \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\") pod \"payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:10.953954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.953924 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.954102 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.954000 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4v6\" (UniqueName: \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.954615 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.954596 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.978525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.978495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4v6\" (UniqueName: \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\") pod \"payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:10.982328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:10.982300 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:11.148329 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.148292 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:11.283461 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.283430 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:11.284945 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:11.284923 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c277ae0_4177_4c37_a3ed_f003d7d04d07.slice/crio-00ffc23189e373751226cd025d22fd018c7e2dbf8e8ff3a7de0350fe25a1aae0 WatchSource:0}: Error finding container 00ffc23189e373751226cd025d22fd018c7e2dbf8e8ff3a7de0350fe25a1aae0: Status 404 returned error can't find the container with id 00ffc23189e373751226cd025d22fd018c7e2dbf8e8ff3a7de0350fe25a1aae0 Oct 29 21:53:11.460869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.460842 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:11.462338 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:11.462311 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147ad627_6596_4419_87f6_ac628dd1d64e.slice/crio-fe7cebdb191417d778f86e23c77bc744613a7a1e6f6e0b358160d86ff2036ab0 WatchSource:0}: Error finding container fe7cebdb191417d778f86e23c77bc744613a7a1e6f6e0b358160d86ff2036ab0: Status 404 returned error can't find the container with id fe7cebdb191417d778f86e23c77bc744613a7a1e6f6e0b358160d86ff2036ab0 Oct 29 21:53:11.642586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.642506 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:11.646185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.646157 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.666016 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.665847 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-044e79d4-serving-cert\"" Oct 29 21:53:11.668272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.668243 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:11.762741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.762711 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgk26\" (UniqueName: \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.762970 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.762769 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.762970 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.762936 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.763089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.763002 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.864110 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.864072 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgk26\" (UniqueName: \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.864293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.864140 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.864293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.864198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.864293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.864239 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.865074 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.865049 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.867101 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.867080 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.867212 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.867117 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.889929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.889869 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgk26\" (UniqueName: \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\") pod \"maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:11.959894 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:11.959858 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:12.171377 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.171338 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" event={"ID":"147ad627-6596-4419-87f6-ac628dd1d64e","Type":"ContainerStarted","Data":"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c"} Oct 29 21:53:12.171572 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.171409 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:12.171572 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.171437 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" event={"ID":"147ad627-6596-4419-87f6-ac628dd1d64e","Type":"ContainerStarted","Data":"fe7cebdb191417d778f86e23c77bc744613a7a1e6f6e0b358160d86ff2036ab0"} Oct 29 21:53:12.172931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.172880 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" event={"ID":"0c277ae0-4177-4c37-a3ed-f003d7d04d07","Type":"ContainerStarted","Data":"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1"} Oct 29 21:53:12.173028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.172939 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" event={"ID":"0c277ae0-4177-4c37-a3ed-f003d7d04d07","Type":"ContainerStarted","Data":"00ffc23189e373751226cd025d22fd018c7e2dbf8e8ff3a7de0350fe25a1aae0"} Oct 29 21:53:12.173087 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.173076 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:12.214637 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.214527 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" podStartSLOduration=2.214513843 podStartE2EDuration="2.214513843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:12.214168535 +0000 UTC m=+651.062645394" watchObservedRunningTime="2026-10-29 21:53:12.214513843 +0000 UTC m=+651.062990689" Oct 29 21:53:12.264057 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.264003 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" podStartSLOduration=2.26398829 podStartE2EDuration="2.26398829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:12.262791511 +0000 UTC m=+651.111268358" watchObservedRunningTime="2026-10-29 21:53:12.26398829 +0000 UTC m=+651.112465136" Oct 29 21:53:12.269015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.268984 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:12.272793 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:12.272769 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab80813_d10a_4ba7_9b55_042d23be9a8e.slice/crio-b14a9d316c8842a6955071d4bdc605a03cf362a1863cb0e30b3281da08532fa4 WatchSource:0}: Error finding container b14a9d316c8842a6955071d4bdc605a03cf362a1863cb0e30b3281da08532fa4: Status 404 returned error can't find the container with id b14a9d316c8842a6955071d4bdc605a03cf362a1863cb0e30b3281da08532fa4 Oct 29 21:53:12.274162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:12.274145 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:53:13.179356 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.179323 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" event={"ID":"cab80813-d10a-4ba7-9b55-042d23be9a8e","Type":"ContainerStarted","Data":"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f"} Oct 29 21:53:13.179793 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.179383 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" event={"ID":"cab80813-d10a-4ba7-9b55-042d23be9a8e","Type":"ContainerStarted","Data":"b14a9d316c8842a6955071d4bdc605a03cf362a1863cb0e30b3281da08532fa4"} Oct 29 21:53:13.227429 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.227373 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" podStartSLOduration=1.875278963 podStartE2EDuration="2.227357106s" totalImagesPullingTime="352.078143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:13.225752075 +0000 UTC m=+652.074228957" watchObservedRunningTime="2026-10-29 21:53:13.227357106 +0000 UTC m=+652.075833951" Oct 29 21:53:13.447389 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.447297 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:13.451422 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.451395 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.481787 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.481752 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:13.491884 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.491847 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.492079 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.491936 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.492154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.492095 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.492223 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.492205 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6cd\" (UniqueName: \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.525023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.524987 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-229ab2ac-serving-cert\"" Oct 29 21:53:13.592771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.592741 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.592974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.592798 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.592974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.592846 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6cd\" (UniqueName: \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.592974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.592917 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.593597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.593572 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.595339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.595306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.595533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.595461 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.617919 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.617877 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6cd\" (UniqueName: \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\") pod \"maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:13.766947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:13.765189 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:14.116931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:14.116871 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:14.119626 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:14.119599 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d65df4f_176f_4369_b9a7_ff5bdd3e32f8.slice/crio-c7ba2aa26f026dcd2529a6035d049e36357484b565e3924dc7ff2a88ccbc439b WatchSource:0}: Error finding container c7ba2aa26f026dcd2529a6035d049e36357484b565e3924dc7ff2a88ccbc439b: Status 404 returned error can't find the container with id c7ba2aa26f026dcd2529a6035d049e36357484b565e3924dc7ff2a88ccbc439b Oct 29 21:53:14.185724 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:14.185686 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" event={"ID":"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8","Type":"ContainerStarted","Data":"c7ba2aa26f026dcd2529a6035d049e36357484b565e3924dc7ff2a88ccbc439b"} Oct 29 21:53:14.186202 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:14.185836 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:15.192447 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:15.192412 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" event={"ID":"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8","Type":"ContainerStarted","Data":"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c"} Oct 29 21:53:15.238924 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:15.238861 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" podStartSLOduration=1.927760978 podStartE2EDuration="2.238844575s" totalImagesPullingTime="311.083597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:15.237673132 +0000 UTC m=+654.086149978" watchObservedRunningTime="2026-10-29 21:53:15.238844575 +0000 UTC m=+654.087321421" Oct 29 21:53:16.206090 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:16.206056 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:16.440928 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:16.438119 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:53:17.262931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.262851 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 21:53:17.267984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.267953 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.286355 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.286314 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f3b067-serving-cert\"" Oct 29 21:53:17.290020 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.289992 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 21:53:17.370921 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.370863 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.371094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.371003 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.371094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.371079 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwgk\" (UniqueName: \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.371201 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.371186 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.472749 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.472712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.472992 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.472774 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.472992 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.472812 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwgk\" (UniqueName: \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.472992 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.472857 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.473586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.473524 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.475445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.475416 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.475520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.475467 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.497241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.497213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwgk\" (UniqueName: \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\") pod \"maas-api-e2e-worker-w0-f3b067-747d944689-msq2v\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:17.589030 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:17.588936 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:18.030704 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:18.030680 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 21:53:18.032688 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:18.032650 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bc08dc_cb42_46aa_ac40_ba7c70701234.slice/crio-a82e6ca8fafa747bce9458fdc7362451afaa06cfb18d4771e0a9ad64dbf97bcc WatchSource:0}: Error finding container a82e6ca8fafa747bce9458fdc7362451afaa06cfb18d4771e0a9ad64dbf97bcc: Status 404 returned error can't find the container with id a82e6ca8fafa747bce9458fdc7362451afaa06cfb18d4771e0a9ad64dbf97bcc Oct 29 21:53:18.215343 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:18.215308 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" event={"ID":"30bc08dc-cb42-46aa-ac40-ba7c70701234","Type":"ContainerStarted","Data":"a82e6ca8fafa747bce9458fdc7362451afaa06cfb18d4771e0a9ad64dbf97bcc"} Oct 29 21:53:19.201642 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.201605 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:19.205487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.205470 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.224439 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.224400 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" event={"ID":"30bc08dc-cb42-46aa-ac40-ba7c70701234","Type":"ContainerStarted","Data":"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771"} Oct 29 21:53:19.224580 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.224551 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:19.231403 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.231378 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2976cd0d-serving-cert\"" Oct 29 21:53:19.263929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.263369 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:19.305165 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.305107 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" podStartSLOduration=1.94263678 podStartE2EDuration="2.305087726s" totalImagesPullingTime="362.450946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:19.303878363 +0000 UTC m=+658.152355213" watchObservedRunningTime="2026-10-29 21:53:19.305087726 +0000 UTC m=+658.153564575" Oct 29 21:53:19.310506 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.310469 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb6td\" (UniqueName: \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.310669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.310519 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.310669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.310648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.310800 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.310702 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.411440 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.411408 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6td\" (UniqueName: \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.411622 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.411448 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.411622 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.411500 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.411622 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.411550 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.412348 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.412321 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.413996 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.413974 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.414135 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.414114 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.459927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.459820 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6td\" (UniqueName: \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\") pod \"maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.518613 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.518581 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:19.864464 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:19.864382 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:19.869004 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:19.868966 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72606b98_5be5_40d5_a5a0_9cd06d842a0d.slice/crio-d4f91ec75b81bdb63b932b0ad17da7c84fb103bd074b7da46936161771d0ebfe WatchSource:0}: Error finding container d4f91ec75b81bdb63b932b0ad17da7c84fb103bd074b7da46936161771d0ebfe: Status 404 returned error can't find the container with id d4f91ec75b81bdb63b932b0ad17da7c84fb103bd074b7da46936161771d0ebfe Oct 29 21:53:20.199683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:20.199659 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:20.230410 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:20.230378 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" event={"ID":"72606b98-5be5-40d5-a5a0-9cd06d842a0d","Type":"ContainerStarted","Data":"d4f91ec75b81bdb63b932b0ad17da7c84fb103bd074b7da46936161771d0ebfe"} Oct 29 21:53:21.081083 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.081051 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:53:21.085149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.085126 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.108993 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.108970 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5d56935a-serving-cert\"" Oct 29 21:53:21.118514 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.118487 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:53:21.168599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.168563 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.168599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.168602 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztr8\" (UniqueName: \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.168866 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.168839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.168954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.168890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.272929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.272767 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.272929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.272823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.272929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.272875 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.272929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.272924 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztr8\" (UniqueName: \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.286002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.274821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.296166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.295431 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.301705 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.301673 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5d56935a-serving-cert\"" Oct 29 21:53:21.302127 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.302108 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:21.302220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.302153 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" event={"ID":"72606b98-5be5-40d5-a5a0-9cd06d842a0d","Type":"ContainerStarted","Data":"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b"} Oct 29 21:53:21.302873 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.302856 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztr8\" (UniqueName: \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.310172 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.310144 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\") pod \"maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.364985 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.363646 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" podStartSLOduration=2.047351667 podStartE2EDuration="2.363629004s" totalImagesPullingTime="316.277337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:21.353230448 +0000 UTC m=+660.201707295" watchObservedRunningTime="2026-10-29 21:53:21.363629004 +0000 UTC m=+660.212105851" Oct 29 21:53:21.398587 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.398555 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:21.733112 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:21.733084 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:53:21.734862 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:21.734831 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c563378_6985_4346_b465_03eef3241536.slice/crio-44a146695381a9409ce59107e1ee09d18557c2d74ee5c5ba0bd040180ef0f7f3 WatchSource:0}: Error finding container 44a146695381a9409ce59107e1ee09d18557c2d74ee5c5ba0bd040180ef0f7f3: Status 404 returned error can't find the container with id 44a146695381a9409ce59107e1ee09d18557c2d74ee5c5ba0bd040180ef0f7f3 Oct 29 21:53:22.245229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:22.245123 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:22.314672 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:22.314629 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" event={"ID":"6c563378-6985-4346-b465-03eef3241536","Type":"ContainerStarted","Data":"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc"} Oct 29 21:53:22.315270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:22.314697 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" event={"ID":"6c563378-6985-4346-b465-03eef3241536","Type":"ContainerStarted","Data":"44a146695381a9409ce59107e1ee09d18557c2d74ee5c5ba0bd040180ef0f7f3"} Oct 29 21:53:22.315455 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:22.315429 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:22.359954 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:22.359856 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" podStartSLOduration=0.953088312 podStartE2EDuration="1.359829619s" totalImagesPullingTime="406.741307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:22.359004076 +0000 UTC m=+661.207480923" watchObservedRunningTime="2026-10-29 21:53:22.359829619 +0000 UTC m=+661.208306466" Oct 29 21:53:23.182401 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:23.182371 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:23.182698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:23.182436 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:24.186092 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:24.186053 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:53:24.186517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:24.186123 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:53:25.255927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:25.253075 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 21:53:26.422721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.422681 4376 generic.go:356] "Generic (PLEG): container finished" podID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerID="07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b" exitCode=0 Oct 29 21:53:26.423235 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.422759 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerDied","Data":"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b"} Oct 29 21:53:26.423319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.423252 4376 scope.go:122] "RemoveContainer" containerID="07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b" Oct 29 21:53:26.425236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.425199 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:26.425593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.425300 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:26.425593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:26.425392 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:27.323065 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:27.322577 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:27.430994 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:27.430955 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerStarted","Data":"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b"} Oct 29 21:53:28.297935 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:28.294437 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 29 21:53:28.297935 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:28.294565 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 29 21:53:28.340162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:28.340128 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:28.412375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:28.412342 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:53:29.308928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:29.308277 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 29 21:53:29.308928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:29.308347 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 29 21:53:30.287043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:30.286960 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 29 21:53:30.287043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:30.287029 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 29 21:53:31.287027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.286990 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 29 21:53:31.287503 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.287040 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 29 21:53:31.836300 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.833294 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 21:53:31.837502 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.837479 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:31.879244 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.879205 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 21:53:31.880797 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.880770 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-29c2f2-serving-cert\"" Oct 29 21:53:31.958962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.958633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:31.958962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.958727 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhtr\" (UniqueName: \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:31.958962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.958765 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:31.958962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:31.958865 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.070467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.069678 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.070467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.069797 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.070467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.069873 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhtr\" (UniqueName: \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.070467 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.069926 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.070823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.070626 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.082943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.082862 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.083468 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.083441 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.098480 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.098409 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhtr\" (UniqueName: \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\") pod \"maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.161941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.161890 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:32.294941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.294415 4376 patch_prober.go:29] interesting pod/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 29 21:53:32.294941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.294476 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 29 21:53:32.515841 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:32.515802 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 21:53:32.522295 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:32.522259 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842bac1d_a464_4937_8668_72bb7966e6e0.slice/crio-c2343ae96b71143f0821b5bf488d4e17940bec326cb2f0362b740db15d453f36 WatchSource:0}: Error finding container c2343ae96b71143f0821b5bf488d4e17940bec326cb2f0362b740db15d453f36: Status 404 returned error can't find the container with id c2343ae96b71143f0821b5bf488d4e17940bec326cb2f0362b740db15d453f36 Oct 29 21:53:33.291659 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.291629 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:33.292160 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.292140 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:33.293041 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.293020 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:33.491073 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.491028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" event={"ID":"842bac1d-a464-4937-8668-72bb7966e6e0","Type":"ContainerStarted","Data":"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1"} Oct 29 21:53:33.491481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.491116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" event={"ID":"842bac1d-a464-4937-8668-72bb7966e6e0","Type":"ContainerStarted","Data":"c2343ae96b71143f0821b5bf488d4e17940bec326cb2f0362b740db15d453f36"} Oct 29 21:53:33.491481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.491301 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:33.537756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:33.537687 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" podStartSLOduration=2.206473473 podStartE2EDuration="2.537668239s" totalImagesPullingTime="331.194766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:33.535482575 +0000 UTC m=+672.383959422" watchObservedRunningTime="2026-10-29 21:53:33.537668239 +0000 UTC m=+672.386145086" Oct 29 21:53:34.497803 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.497769 4376 generic.go:356] "Generic (PLEG): container finished" podID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerID="e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf" exitCode=0 Oct 29 21:53:34.498258 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.497839 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerDied","Data":"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf"} Oct 29 21:53:34.498563 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.498544 4376 scope.go:122] "RemoveContainer" containerID="e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf" Oct 29 21:53:34.499815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.499789 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:34.499864 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.499855 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:34.499899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:34.499885 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:35.530927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.530221 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerStarted","Data":"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140"} Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.536550 4376 generic.go:356] "Generic (PLEG): container finished" podID="64cdab92-ac02-434a-97fd-d6df904b2365" containerID="d1ecc8e5a9251cdc26519b74a76428c5d0bbe202185efe2c5830e282b782d269" exitCode=0 Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.536637 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerDied","Data":"d1ecc8e5a9251cdc26519b74a76428c5d0bbe202185efe2c5830e282b782d269"} Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.537127 4376 scope.go:122] "RemoveContainer" containerID="d1ecc8e5a9251cdc26519b74a76428c5d0bbe202185efe2c5830e282b782d269" Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.543484 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.543578 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:35.543931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:35.543629 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:36.137199 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:36.137167 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:36.137597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:36.137569 4376 patch_prober.go:29] interesting pod/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 21:53:36.137795 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:36.137766 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 21:53:36.565446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:36.564540 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerStarted","Data":"a2d614f5007a4b94982331fa2fb43010e0e02ff8ca04228ba70658ff89d93d4e"} Oct 29 21:53:37.062379 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:37.062295 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:37.070429 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:37.070051 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 29 21:53:37.070429 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:37.070116 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 29 21:53:37.137748 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:37.137312 4376 patch_prober.go:29] interesting pod/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 21:53:37.137748 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:37.137378 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 21:53:38.065359 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:38.065318 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 29 21:53:38.066016 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:38.065984 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 29 21:53:38.148928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:38.147016 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:38.148928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:38.148246 4376 patch_prober.go:29] interesting pod/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 21:53:38.148928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:38.148306 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 21:53:39.062554 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:39.062506 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 29 21:53:39.062881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:39.062833 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 29 21:53:39.149361 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:39.149321 4376 patch_prober.go:29] interesting pod/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 21:53:39.149963 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:39.149925 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 21:53:39.522897 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:39.522867 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 21:53:40.065387 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.065344 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 29 21:53:40.065687 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.065655 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 29 21:53:40.156094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.156056 4376 patch_prober.go:29] interesting pod/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 21:53:40.156771 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.156739 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 21:53:40.156932 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.156681 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:40.601322 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.601289 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:40.601772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.601738 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" podUID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" containerName="maas-api" containerID="cri-o://d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c" gracePeriod=30 Oct 29 21:53:40.785579 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.785544 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:40.802084 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:40.801668 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:41.046367 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.043142 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:41.065438 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.065015 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 29 21:53:41.065438 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.065077 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 29 21:53:41.166198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.166145 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\") pod \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " Oct 29 21:53:41.167388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.167359 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\" (UniqueName: \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\") pod \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " Oct 29 21:53:41.167618 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.167597 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\") pod \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " Oct 29 21:53:41.167767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.167750 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\") pod \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\" (UID: \"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8\") " Oct 29 21:53:41.180181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.180114 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca" pod "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" (UID: "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:41.200172 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.187132 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls" pod "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" (UID: "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:41.200172 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.187304 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs" pod "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" (UID: "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:41.207357 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.207315 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd" pod "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" (UID: "6d65df4f-176f-4369-b9a7-ff5bdd3e32f8"). InnerVolumeSpecName "kube-api-access-zz6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:41.289226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.288989 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6cd\" (UniqueName: \"kubernetes.io/projected/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-kube-api-access-zz6cd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:41.289445 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.289428 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:41.289556 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.289543 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:41.289635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.289625 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:41.381501 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.380456 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:41.384623 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.384588 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:53:41.624255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.624107 4376 generic.go:356] "Generic (PLEG): container finished" podID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" containerID="d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c" exitCode=0 Oct 29 21:53:41.624255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.624195 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" event={"ID":"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8","Type":"ContainerDied","Data":"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c"} Oct 29 21:53:41.624487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.624283 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" event={"ID":"6d65df4f-176f-4369-b9a7-ff5bdd3e32f8","Type":"ContainerDied","Data":"c7ba2aa26f026dcd2529a6035d049e36357484b565e3924dc7ff2a88ccbc439b"} Oct 29 21:53:41.624487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.624225 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm" Oct 29 21:53:41.624487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.624325 4376 scope.go:122] "RemoveContainer" containerID="d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c" Oct 29 21:53:41.659145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.659105 4376 scope.go:122] "RemoveContainer" containerID="d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c" Oct 29 21:53:41.659732 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:41.659702 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c\": container with ID starting with d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c not found: ID does not exist" containerID="d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c" Oct 29 21:53:41.659847 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.659743 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c"} err="failed to get container status \"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c\": rpc error: code = NotFound desc = could not find container \"d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c\": container with ID starting with d08d0842a416fa58930d66c623ad75c67f3b0c7eba626c0a53dbd26a142dc44c not found: ID does not exist" Oct 29 21:53:41.682542 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.682512 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:41.688046 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:41.688023 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229ab2ac-6f69cd944f-9qspm"] Oct 29 21:53:42.127728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:42.127694 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:42.129400 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:42.129373 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 21:53:43.272560 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:43.272521 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" path="/var/lib/kubelet/pods/6d65df4f-176f-4369-b9a7-ff5bdd3e32f8/volumes" Oct 29 21:53:45.785856 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:45.785793 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" podUID="147ad627-6596-4419-87f6-ac628dd1d64e" containerName="payload-processing" containerID="cri-o://577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c" gracePeriod=25 Oct 29 21:53:45.802489 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:45.802454 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" podUID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" containerName="payload-pre-processing" containerID="cri-o://7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1" gracePeriod=25 Oct 29 21:53:46.065767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.065743 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:46.157155 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.157123 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:46.167296 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.167261 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\") pod \"147ad627-6596-4419-87f6-ac628dd1d64e\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " Oct 29 21:53:46.167605 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.167587 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\" (UniqueName: \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\") pod \"147ad627-6596-4419-87f6-ac628dd1d64e\" (UID: \"147ad627-6596-4419-87f6-ac628dd1d64e\") " Oct 29 21:53:46.167974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.167885 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume" pod "147ad627-6596-4419-87f6-ac628dd1d64e" (UID: "147ad627-6596-4419-87f6-ac628dd1d64e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:46.168184 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.168163 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147ad627-6596-4419-87f6-ac628dd1d64e-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:46.172884 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.172857 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6" pod "147ad627-6596-4419-87f6-ac628dd1d64e" (UID: "147ad627-6596-4419-87f6-ac628dd1d64e"). InnerVolumeSpecName "kube-api-access-tr4v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:46.269055 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.269023 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\" (UniqueName: \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\") pod \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " Oct 29 21:53:46.269220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.269133 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\") pod \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\" (UID: \"0c277ae0-4177-4c37-a3ed-f003d7d04d07\") " Oct 29 21:53:46.269393 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.269380 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr4v6\" (UniqueName: \"kubernetes.io/projected/147ad627-6596-4419-87f6-ac628dd1d64e-kube-api-access-tr4v6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:46.269463 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.269439 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume" pod "0c277ae0-4177-4c37-a3ed-f003d7d04d07" (UID: "0c277ae0-4177-4c37-a3ed-f003d7d04d07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:46.271067 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.271047 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4" pod "0c277ae0-4177-4c37-a3ed-f003d7d04d07" (UID: "0c277ae0-4177-4c37-a3ed-f003d7d04d07"). InnerVolumeSpecName "kube-api-access-9wgs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:46.370622 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.370539 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c277ae0-4177-4c37-a3ed-f003d7d04d07-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:46.370622 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.370566 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgs4\" (UniqueName: \"kubernetes.io/projected/0c277ae0-4177-4c37-a3ed-f003d7d04d07-kube-api-access-9wgs4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:46.433528 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:46.433481 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:53:46.692349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.692230 4376 generic.go:356] "Generic (PLEG): container finished" podID="147ad627-6596-4419-87f6-ac628dd1d64e" containerID="577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c" exitCode=0 Oct 29 21:53:46.692565 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.692416 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" Oct 29 21:53:46.694438 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.692937 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" event={"ID":"147ad627-6596-4419-87f6-ac628dd1d64e","Type":"ContainerDied","Data":"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c"} Oct 29 21:53:46.694438 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.693003 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz" event={"ID":"147ad627-6596-4419-87f6-ac628dd1d64e","Type":"ContainerDied","Data":"fe7cebdb191417d778f86e23c77bc744613a7a1e6f6e0b358160d86ff2036ab0"} Oct 29 21:53:46.694438 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.693035 4376 scope.go:122] "RemoveContainer" containerID="577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c" Oct 29 21:53:46.704128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.696410 4376 generic.go:356] "Generic (PLEG): container finished" podID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" containerID="7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1" exitCode=0 Oct 29 21:53:46.704128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.696558 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" event={"ID":"0c277ae0-4177-4c37-a3ed-f003d7d04d07","Type":"ContainerDied","Data":"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1"} Oct 29 21:53:46.704128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.696596 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" event={"ID":"0c277ae0-4177-4c37-a3ed-f003d7d04d07","Type":"ContainerDied","Data":"00ffc23189e373751226cd025d22fd018c7e2dbf8e8ff3a7de0350fe25a1aae0"} Oct 29 21:53:46.704128 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.696656 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l" Oct 29 21:53:46.708226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.708197 4376 scope.go:122] "RemoveContainer" containerID="577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c" Oct 29 21:53:46.708642 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:46.708612 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c\": container with ID starting with 577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c not found: ID does not exist" containerID="577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c" Oct 29 21:53:46.708787 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.708654 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c"} err="failed to get container status \"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c\": rpc error: code = NotFound desc = could not find container \"577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c\": container with ID starting with 577c6d71c0d2c3a0a0731a67a3d84aaac245bb748b649a3cacfdac95008c507c not found: ID does not exist" Oct 29 21:53:46.708787 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.708680 4376 scope.go:122] "RemoveContainer" containerID="7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1" Oct 29 21:53:46.720087 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.720065 4376 scope.go:122] "RemoveContainer" containerID="7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1" Oct 29 21:53:46.720412 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:46.720395 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1\": container with ID starting with 7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1 not found: ID does not exist" containerID="7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1" Oct 29 21:53:46.720453 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.720420 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1"} err="failed to get container status \"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1\": rpc error: code = NotFound desc = could not find container \"7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1\": container with ID starting with 7d0450f6a998d5fc1421b0cda237297bdac98ea88da2ccbb6f536d8c1883f8b1 not found: ID does not exist" Oct 29 21:53:46.764569 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.764411 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:46.776581 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.776544 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229ab2ac-6ccdcdfdfd-t42kz"] Oct 29 21:53:46.804920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.804877 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:46.814588 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:46.814551 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229ab2ac-cdb879dd7-8vk4l"] Oct 29 21:53:47.272106 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:47.272072 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" path="/var/lib/kubelet/pods/0c277ae0-4177-4c37-a3ed-f003d7d04d07/volumes" Oct 29 21:53:47.272478 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:47.272465 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147ad627-6596-4419-87f6-ac628dd1d64e" path="/var/lib/kubelet/pods/147ad627-6596-4419-87f6-ac628dd1d64e/volumes" Oct 29 21:53:51.806716 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:51.806664 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:51.807301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:51.806989 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" podUID="cab80813-d10a-4ba7-9b55-042d23be9a8e" containerName="maas-api" containerID="cri-o://c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f" gracePeriod=30 Oct 29 21:53:52.152198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.152160 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:52.264983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.264946 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\" (UniqueName: \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\") pod \"cab80813-d10a-4ba7-9b55-042d23be9a8e\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " Oct 29 21:53:52.265220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.265003 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\") pod \"cab80813-d10a-4ba7-9b55-042d23be9a8e\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " Oct 29 21:53:52.265220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.265115 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\") pod \"cab80813-d10a-4ba7-9b55-042d23be9a8e\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " Oct 29 21:53:52.265220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.265152 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\") pod \"cab80813-d10a-4ba7-9b55-042d23be9a8e\" (UID: \"cab80813-d10a-4ba7-9b55-042d23be9a8e\") " Oct 29 21:53:52.265508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.265484 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca" pod "cab80813-d10a-4ba7-9b55-042d23be9a8e" (UID: "cab80813-d10a-4ba7-9b55-042d23be9a8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:52.267526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.267488 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls" pod "cab80813-d10a-4ba7-9b55-042d23be9a8e" (UID: "cab80813-d10a-4ba7-9b55-042d23be9a8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:52.267649 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.267557 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs" pod "cab80813-d10a-4ba7-9b55-042d23be9a8e" (UID: "cab80813-d10a-4ba7-9b55-042d23be9a8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:52.267649 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.267556 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26" pod "cab80813-d10a-4ba7-9b55-042d23be9a8e" (UID: "cab80813-d10a-4ba7-9b55-042d23be9a8e"). InnerVolumeSpecName "kube-api-access-hgk26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:52.366154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.366113 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgk26\" (UniqueName: \"kubernetes.io/projected/cab80813-d10a-4ba7-9b55-042d23be9a8e-kube-api-access-hgk26\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:52.366154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.366148 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:52.366154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.366165 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab80813-d10a-4ba7-9b55-042d23be9a8e-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:52.366455 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.366174 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab80813-d10a-4ba7-9b55-042d23be9a8e-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:52.667764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.667724 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:52.668104 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.668073 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" containerID="cri-o://1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b" gracePeriod=30 Oct 29 21:53:52.735998 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.735958 4376 generic.go:356] "Generic (PLEG): container finished" podID="cab80813-d10a-4ba7-9b55-042d23be9a8e" containerID="c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f" exitCode=0 Oct 29 21:53:52.736190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.736032 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" Oct 29 21:53:52.736190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.736052 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" event={"ID":"cab80813-d10a-4ba7-9b55-042d23be9a8e","Type":"ContainerDied","Data":"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f"} Oct 29 21:53:52.736190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.736104 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8" event={"ID":"cab80813-d10a-4ba7-9b55-042d23be9a8e","Type":"ContainerDied","Data":"b14a9d316c8842a6955071d4bdc605a03cf362a1863cb0e30b3281da08532fa4"} Oct 29 21:53:52.736190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.736126 4376 scope.go:122] "RemoveContainer" containerID="c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f" Oct 29 21:53:52.745949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.745928 4376 scope.go:122] "RemoveContainer" containerID="c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f" Oct 29 21:53:52.746199 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:52.746181 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f\": container with ID starting with c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f not found: ID does not exist" containerID="c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f" Oct 29 21:53:52.746260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.746213 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f"} err="failed to get container status \"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f\": rpc error: code = NotFound desc = could not find container \"c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f\": container with ID starting with c39c0fc6d8db11638dd64480ba6cca047712e59ccb7cc488730df695a2489f6f not found: ID does not exist" Oct 29 21:53:52.791867 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.791824 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:52.795192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:52.795166 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-044e79d4-5b74699fd8-54bl8"] Oct 29 21:53:53.272102 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.272071 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab80813-d10a-4ba7-9b55-042d23be9a8e" path="/var/lib/kubelet/pods/cab80813-d10a-4ba7-9b55-042d23be9a8e/volumes" Oct 29 21:53:53.983136 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.983094 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 21:53:53.983915 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.983878 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" containerName="maas-api" Oct 29 21:53:53.984008 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.983919 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" containerName="maas-api" Oct 29 21:53:53.984008 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.983984 4376 cpu_manager.go:414] "containerMap: removing container" podUID="147ad627-6596-4419-87f6-ac628dd1d64e" containerName="payload-processing" Oct 29 21:53:53.984008 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.983993 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147ad627-6596-4419-87f6-ac628dd1d64e" containerName="payload-processing" Oct 29 21:53:53.984008 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984004 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cab80813-d10a-4ba7-9b55-042d23be9a8e" containerName="maas-api" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984013 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab80813-d10a-4ba7-9b55-042d23be9a8e" containerName="maas-api" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984034 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" containerName="payload-pre-processing" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984042 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" containerName="payload-pre-processing" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984182 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="147ad627-6596-4419-87f6-ac628dd1d64e" containerName="payload-processing" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984193 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab80813-d10a-4ba7-9b55-042d23be9a8e" containerName="maas-api" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984204 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c277ae0-4177-4c37-a3ed-f003d7d04d07" containerName="payload-pre-processing" Oct 29 21:53:53.984229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.984217 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d65df4f-176f-4369-b9a7-ff5bdd3e32f8" containerName="maas-api" Oct 29 21:53:53.987793 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:53.987766 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.009762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.009732 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 21:53:54.018970 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.018933 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-92d984-openshift-default-dockercfg-5jl7t\"" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120583 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120685 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnmj\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120856 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120879 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120926 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.125928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.120963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222636 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222598 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnmj\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222676 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222717 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222753 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222789 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.222839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222814 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.222845 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.223162 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223340 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.223247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223372 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.223351 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.223520 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.223547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.223541 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.225147 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.225120 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.225327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.225311 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.251008 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.249292 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnmj\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.263953 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.254237 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.303629 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.303591 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:54.624616 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.624588 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 21:53:54.626204 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:54.626175 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17317361_6678_4683_9202_e15636063a6b.slice/crio-d7904de52b2f04f1d5b8fb0dfb1506332dfee9fbfa3285b66f5250ee54a9f51a WatchSource:0}: Error finding container d7904de52b2f04f1d5b8fb0dfb1506332dfee9fbfa3285b66f5250ee54a9f51a: Status 404 returned error can't find the container with id d7904de52b2f04f1d5b8fb0dfb1506332dfee9fbfa3285b66f5250ee54a9f51a Oct 29 21:53:54.628528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.628495 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:54.628597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.628563 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:54.628633 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.628595 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:53:54.750949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.750876 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerStarted","Data":"5b6b270d46863b583ed577d3f86f979613b612771a08acd6fa13632aa0ae45b9"} Oct 29 21:53:54.751233 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.750973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerStarted","Data":"d7904de52b2f04f1d5b8fb0dfb1506332dfee9fbfa3285b66f5250ee54a9f51a"} Oct 29 21:53:54.798962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:54.798897 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podStartSLOduration=1.798881634 podStartE2EDuration="1.798881634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:54.796922951 +0000 UTC m=+693.645399837" watchObservedRunningTime="2026-10-29 21:53:54.798881634 +0000 UTC m=+693.647358479" Oct 29 21:53:55.304258 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:55.304219 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:55.310037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:55.310007 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:55.756987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:55.756957 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:55.757865 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:55.757847 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:53:57.927196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:57.927173 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:58.064060 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.064027 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:58.064307 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.064280 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" podUID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" containerName="maas-api" containerID="cri-o://93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b" gracePeriod=30 Oct 29 21:53:58.072224 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072196 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072244 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072283 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072327 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072510 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072353 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072510 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072462 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072684 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072574 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072684 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072642 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.072862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.072680 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\") pod \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\" (UID: \"a70f06ff-f5fc-45bf-bfe8-c672404e97da\") " Oct 29 21:53:58.074120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.073511 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:58.074120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.073536 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:58.074120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.073731 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:58.074120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.073751 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:58.074841 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.074804 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:58.076487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.076459 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 21:53:58.077218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.077191 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:58.078341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.078312 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "kube-api-access-zsg4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:58.078441 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.078422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token" pod "a70f06ff-f5fc-45bf-bfe8-c672404e97da" (UID: "a70f06ff-f5fc-45bf-bfe8-c672404e97da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:58.174535 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174498 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsg4w\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-kube-api-access-zsg4w\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174535 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174539 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174553 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174567 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174577 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174584 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174595 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174603 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.174762 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.174610 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70f06ff-f5fc-45bf-bfe8-c672404e97da-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.341970 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.341940 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:58.477818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.477779 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\" (UniqueName: \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\") pod \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " Oct 29 21:53:58.478032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.477850 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\") pod \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " Oct 29 21:53:58.478032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.477959 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\") pod \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " Oct 29 21:53:58.478162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.478055 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\") pod \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\" (UID: \"72606b98-5be5-40d5-a5a0-9cd06d842a0d\") " Oct 29 21:53:58.478382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.478348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca" pod "72606b98-5be5-40d5-a5a0-9cd06d842a0d" (UID: "72606b98-5be5-40d5-a5a0-9cd06d842a0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:58.478523 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.478464 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72606b98-5be5-40d5-a5a0-9cd06d842a0d-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.480068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.480039 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs" pod "72606b98-5be5-40d5-a5a0-9cd06d842a0d" (UID: "72606b98-5be5-40d5-a5a0-9cd06d842a0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:58.480068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.480054 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls" pod "72606b98-5be5-40d5-a5a0-9cd06d842a0d" (UID: "72606b98-5be5-40d5-a5a0-9cd06d842a0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:58.480068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.480056 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td" pod "72606b98-5be5-40d5-a5a0-9cd06d842a0d" (UID: "72606b98-5be5-40d5-a5a0-9cd06d842a0d"). InnerVolumeSpecName "kube-api-access-jb6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:58.580449 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.580400 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.580449 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.580454 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb6td\" (UniqueName: \"kubernetes.io/projected/72606b98-5be5-40d5-a5a0-9cd06d842a0d-kube-api-access-jb6td\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.580750 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.580476 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72606b98-5be5-40d5-a5a0-9cd06d842a0d-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.616786 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.616704 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 21:53:58.617198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617183 4376 cpu_manager.go:414] "containerMap: removing container" podUID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" containerName="maas-api" Oct 29 21:53:58.617240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617200 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" containerName="maas-api" Oct 29 21:53:58.617240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617210 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.617240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617215 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.617240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617224 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.617240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617229 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.617420 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617352 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" containerName="maas-api" Oct 29 21:53:58.617420 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617366 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.617420 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.617376 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerName="istio-proxy" Oct 29 21:53:58.619765 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.619747 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.645309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.645270 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-91520c-serving-cert\"" Oct 29 21:53:58.648146 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.648113 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 21:53:58.681237 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.681210 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.681386 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.681243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmnx\" (UniqueName: \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.681386 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.681287 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.681525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.681499 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.774291 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.774255 4376 generic.go:356] "Generic (PLEG): container finished" podID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" containerID="93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b" exitCode=0 Oct 29 21:53:58.774474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.774306 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" Oct 29 21:53:58.774474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.774343 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" event={"ID":"72606b98-5be5-40d5-a5a0-9cd06d842a0d","Type":"ContainerDied","Data":"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b"} Oct 29 21:53:58.774474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.774393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp" event={"ID":"72606b98-5be5-40d5-a5a0-9cd06d842a0d","Type":"ContainerDied","Data":"d4f91ec75b81bdb63b932b0ad17da7c84fb103bd074b7da46936161771d0ebfe"} Oct 29 21:53:58.774474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.774425 4376 scope.go:122] "RemoveContainer" containerID="93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b" Oct 29 21:53:58.776231 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.776207 4376 generic.go:356] "Generic (PLEG): container finished" podID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" containerID="1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b" exitCode=0 Oct 29 21:53:58.776347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.776255 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" Oct 29 21:53:58.776347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.776264 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerDied","Data":"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b"} Oct 29 21:53:58.776347 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.776301 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj" event={"ID":"a70f06ff-f5fc-45bf-bfe8-c672404e97da","Type":"ContainerDied","Data":"38fe489251ca633d4da4fb0109290ab8196a1bee44a67c91d3ff829d8b95d099"} Oct 29 21:53:58.782318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.782288 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.782419 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.782334 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmnx\" (UniqueName: \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.782419 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.782380 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.782541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.782478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.783325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.783287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.784896 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.784721 4376 scope.go:122] "RemoveContainer" containerID="93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b" Oct 29 21:53:58.785005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.784991 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.785071 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:58.785041 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b\": container with ID starting with 93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b not found: ID does not exist" containerID="93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b" Oct 29 21:53:58.785124 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.785080 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b"} err="failed to get container status \"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b\": rpc error: code = NotFound desc = could not find container \"93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b\": container with ID starting with 93082ee34c988fc8773bec0561ec7b540c8aec81ef1c4f9374a161f175578f7b not found: ID does not exist" Oct 29 21:53:58.785124 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.785106 4376 scope.go:122] "RemoveContainer" containerID="1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b" Oct 29 21:53:58.785356 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.785339 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.807437 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.807415 4376 scope.go:122] "RemoveContainer" containerID="07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b" Oct 29 21:53:58.810767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.810742 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmnx\" (UniqueName: \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\") pod \"maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.878403 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.879545 4376 scope.go:122] "RemoveContainer" containerID="1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:58.880116 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b\": container with ID starting with 1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b not found: ID does not exist" containerID="1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.880180 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b"} err="failed to get container status \"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b\": rpc error: code = NotFound desc = could not find container \"1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b\": container with ID starting with 1081ba00c9c945419d6af9580b6f13f1b65e610998dfde3418c2d70562122b4b not found: ID does not exist" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.880225 4376 scope.go:122] "RemoveContainer" containerID="07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:53:58.880586 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b\": container with ID starting with 07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b not found: ID does not exist" containerID="07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b" Oct 29 21:53:58.885597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.880620 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b"} err="failed to get container status \"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b\": rpc error: code = NotFound desc = could not find container \"07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b\": container with ID starting with 07b315eedef3b7a27fb26f0c5a4a0f28078e97e2c27b08fc388764c71989ad5b not found: ID does not exist" Oct 29 21:53:58.900367 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.900328 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-044e79d4-openshift-default-5fb9bbfd4c-lmhzj"] Oct 29 21:53:58.932761 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.932609 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:58.937084 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.937063 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:58.941367 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:58.940637 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2976cd0d-856846dcdf-jrwcp"] Oct 29 21:53:59.253110 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.253074 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 21:53:59.260462 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:53:59.260381 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3551b155_6d88_4b72_865a_02c0c00fe90a.slice/crio-0d67994a74e30307762b519d0d15a6e0ad926f3a3197db62757fb3419b64f534 WatchSource:0}: Error finding container 0d67994a74e30307762b519d0d15a6e0ad926f3a3197db62757fb3419b64f534: Status 404 returned error can't find the container with id 0d67994a74e30307762b519d0d15a6e0ad926f3a3197db62757fb3419b64f534 Oct 29 21:53:59.271365 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.271334 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72606b98-5be5-40d5-a5a0-9cd06d842a0d" path="/var/lib/kubelet/pods/72606b98-5be5-40d5-a5a0-9cd06d842a0d/volumes" Oct 29 21:53:59.271794 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.271773 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70f06ff-f5fc-45bf-bfe8-c672404e97da" path="/var/lib/kubelet/pods/a70f06ff-f5fc-45bf-bfe8-c672404e97da/volumes" Oct 29 21:53:59.381537 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.381504 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:53:59.381791 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.381745 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" containerID="cri-o://71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140" gracePeriod=30 Oct 29 21:53:59.785893 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.785860 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" event={"ID":"3551b155-6d88-4b72-865a-02c0c00fe90a","Type":"ContainerStarted","Data":"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef"} Oct 29 21:53:59.786104 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.785945 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" event={"ID":"3551b155-6d88-4b72-865a-02c0c00fe90a","Type":"ContainerStarted","Data":"0d67994a74e30307762b519d0d15a6e0ad926f3a3197db62757fb3419b64f534"} Oct 29 21:53:59.786104 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.786041 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:53:59.860441 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:53:59.860384 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" podStartSLOduration=1.4597923910000001 podStartE2EDuration="1.860369761s" totalImagesPullingTime="400.57737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:59.859293133 +0000 UTC m=+698.707769979" watchObservedRunningTime="2026-10-29 21:53:59.860369761 +0000 UTC m=+698.708846606" Oct 29 21:54:00.196926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.195964 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:54:00.196926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.196225 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" podUID="6c563378-6985-4346-b465-03eef3241536" containerName="maas-api" containerID="cri-o://aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc" gracePeriod=30 Oct 29 21:54:00.498891 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.498865 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:54:00.530504 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.530471 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\") pod \"6c563378-6985-4346-b465-03eef3241536\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " Oct 29 21:54:00.530677 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.530512 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\") pod \"6c563378-6985-4346-b465-03eef3241536\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " Oct 29 21:54:00.530737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.530721 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\" (UniqueName: \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\") pod \"6c563378-6985-4346-b465-03eef3241536\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " Oct 29 21:54:00.530852 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.530836 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\") pod \"6c563378-6985-4346-b465-03eef3241536\" (UID: \"6c563378-6985-4346-b465-03eef3241536\") " Oct 29 21:54:00.531205 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.531181 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca" pod "6c563378-6985-4346-b465-03eef3241536" (UID: "6c563378-6985-4346-b465-03eef3241536"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:54:00.532716 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.532694 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8" pod "6c563378-6985-4346-b465-03eef3241536" (UID: "6c563378-6985-4346-b465-03eef3241536"). InnerVolumeSpecName "kube-api-access-6ztr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:00.533144 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.533121 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs" pod "6c563378-6985-4346-b465-03eef3241536" (UID: "6c563378-6985-4346-b465-03eef3241536"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:54:00.533144 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.533133 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls" pod "6c563378-6985-4346-b465-03eef3241536" (UID: "6c563378-6985-4346-b465-03eef3241536"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:54:00.632130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.632098 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:00.632130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.632128 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztr8\" (UniqueName: \"kubernetes.io/projected/6c563378-6985-4346-b465-03eef3241536-kube-api-access-6ztr8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:00.632130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.632138 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c563378-6985-4346-b465-03eef3241536-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:00.632379 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.632147 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c563378-6985-4346-b465-03eef3241536-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:00.709573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.709537 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 21:54:00.710436 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.710382 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6c563378-6985-4346-b465-03eef3241536" containerName="maas-api" Oct 29 21:54:00.710436 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.710409 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c563378-6985-4346-b465-03eef3241536" containerName="maas-api" Oct 29 21:54:00.710790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.710646 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c563378-6985-4346-b465-03eef3241536" containerName="maas-api" Oct 29 21:54:00.713645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.713622 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.732986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.732886 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.732998 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5ms\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733140 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733235 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733180 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733287 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733261 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733314 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733434 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733417 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.733480 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.733458 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.735871 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.735815 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-863f1b-openshift-default-dockercfg-5qt7r\"" Oct 29 21:54:00.739766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.739745 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 21:54:00.792314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.792277 4376 generic.go:356] "Generic (PLEG): container finished" podID="6c563378-6985-4346-b465-03eef3241536" containerID="aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc" exitCode=0 Oct 29 21:54:00.792521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.792340 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" Oct 29 21:54:00.792521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.792369 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" event={"ID":"6c563378-6985-4346-b465-03eef3241536","Type":"ContainerDied","Data":"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc"} Oct 29 21:54:00.792521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.792437 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh" event={"ID":"6c563378-6985-4346-b465-03eef3241536","Type":"ContainerDied","Data":"44a146695381a9409ce59107e1ee09d18557c2d74ee5c5ba0bd040180ef0f7f3"} Oct 29 21:54:00.792521 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.792468 4376 scope.go:122] "RemoveContainer" containerID="aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc" Oct 29 21:54:00.803185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.803167 4376 scope.go:122] "RemoveContainer" containerID="aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc" Oct 29 21:54:00.803511 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:00.803486 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc\": container with ID starting with aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc not found: ID does not exist" containerID="aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc" Oct 29 21:54:00.803607 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.803516 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc"} err="failed to get container status \"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc\": rpc error: code = NotFound desc = could not find container \"aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc\": container with ID starting with aec49196115b08043589bb52b0030998b4c7e4ed2780201fa196bbe702d404bc not found: ID does not exist" Oct 29 21:54:00.834528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.834732 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834556 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5ms\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.834811 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834769 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.834870 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834820 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.834958 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834892 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.834958 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834938 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.834968 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835078 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835063 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835249 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835280 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835374 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835448 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.835662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.835620 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.837303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.837284 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.837441 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.837422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.848786 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.848761 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:54:00.853754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.853728 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d56935a-7cb777774c-bs6sh"] Oct 29 21:54:00.859535 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.859515 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5ms\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:00.859725 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:00.859708 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:01.043986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.039114 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:01.272171 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.272132 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c563378-6985-4346-b465-03eef3241536" path="/var/lib/kubelet/pods/6c563378-6985-4346-b465-03eef3241536/volumes" Oct 29 21:54:01.382052 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.382025 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 21:54:01.383872 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:01.383847 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eb8767_d77f_4e57_a298_f7e4e9abe316.slice/crio-b60035ca811289957d4a8db67844ef5ea688b4035a4e08997c4c76fc97b43a27 WatchSource:0}: Error finding container b60035ca811289957d4a8db67844ef5ea688b4035a4e08997c4c76fc97b43a27: Status 404 returned error can't find the container with id b60035ca811289957d4a8db67844ef5ea688b4035a4e08997c4c76fc97b43a27 Oct 29 21:54:01.386306 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.386268 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:01.386421 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.386356 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:01.386421 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.386408 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:01.799839 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.799802 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerStarted","Data":"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562"} Oct 29 21:54:01.800040 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.799873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerStarted","Data":"b60035ca811289957d4a8db67844ef5ea688b4035a4e08997c4c76fc97b43a27"} Oct 29 21:54:01.850105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:01.850048 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podStartSLOduration=1.850033727 podStartE2EDuration="1.850033727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:01.848030227 +0000 UTC m=+700.696507073" watchObservedRunningTime="2026-10-29 21:54:01.850033727 +0000 UTC m=+700.698510652" Oct 29 21:54:02.040181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:02.040138 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:02.045316 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:02.045285 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:02.808270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:02.808241 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:02.809189 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:02.809171 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:03.886874 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.886837 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 21:54:03.890450 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.890419 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897048 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtt6\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897101 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897168 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897285 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897316 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897356 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897529 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897522 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.897690 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.897550 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.917658 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.917620 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 21:54:03.923215 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.923186 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cea94a-openshift-default-dockercfg-56p6l\"" Oct 29 21:54:03.998364 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998324 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998380 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998439 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998464 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998527 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998631 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998657 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.998818 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998711 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtt6\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.999007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998862 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.999007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.998921 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.999145 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.999096 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.999216 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.999195 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:03.999272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:03.999222 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.001065 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.001036 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.001186 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.001125 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.028295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.028257 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.028453 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.028359 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtt6\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.225223 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.225182 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:04.601251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.601217 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 21:54:04.603288 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:04.603256 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d26d931_118e_4738_9aaa_a0074161de15.slice/crio-417a6882ccadce1c83840ea212859e033e5dd7d82ecd7d799b05b17a3cac5a3a WatchSource:0}: Error finding container 417a6882ccadce1c83840ea212859e033e5dd7d82ecd7d799b05b17a3cac5a3a: Status 404 returned error can't find the container with id 417a6882ccadce1c83840ea212859e033e5dd7d82ecd7d799b05b17a3cac5a3a Oct 29 21:54:04.605845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.605800 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:04.605980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.605917 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:04.605980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.605969 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:04.694124 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.694102 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:54:04.705781 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.705747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.705981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.705821 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706054 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706009 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706103 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706062 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706103 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706133 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706181 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706229 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706296 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706234 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\") pod \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\" (UID: \"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a\") " Oct 29 21:54:04.706897 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.706873 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:04.707111 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.707091 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:04.707691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.707558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:04.707820 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.707746 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:04.707928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.707885 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:54:04.710068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.710039 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 21:54:04.710219 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.710073 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "kube-api-access-dr85l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:04.710327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.710306 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:04.710595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.710558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy" pod "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" (UID: "b3b86e2b-8441-4ec1-b19b-b29b06a5f96a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:04.841000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.812945 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr85l\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-kube-api-access-dr85l\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842021 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842054 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842070 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842084 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842105 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842118 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842129 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.843068 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.842142 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:04.859716 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.859681 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerStarted","Data":"db58965ebd58993603641e2682d9ab0691c5f758d4f072466b9875f81982b925"} Oct 29 21:54:04.859956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.859745 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerStarted","Data":"417a6882ccadce1c83840ea212859e033e5dd7d82ecd7d799b05b17a3cac5a3a"} Oct 29 21:54:04.862528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.862498 4376 generic.go:356] "Generic (PLEG): container finished" podID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerID="71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140" exitCode=0 Oct 29 21:54:04.862662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.862558 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" Oct 29 21:54:04.862662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.862582 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerDied","Data":"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140"} Oct 29 21:54:04.862662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.862644 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm" event={"ID":"b3b86e2b-8441-4ec1-b19b-b29b06a5f96a","Type":"ContainerDied","Data":"05b212322d9ccee18ab2543787b71570b934f8dcb8d6eca0d2bd7abe9d5f0cb2"} Oct 29 21:54:04.862760 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.862667 4376 scope.go:122] "RemoveContainer" containerID="71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140" Oct 29 21:54:04.872816 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.872780 4376 scope.go:122] "RemoveContainer" containerID="e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf" Oct 29 21:54:04.884769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.884746 4376 scope.go:122] "RemoveContainer" containerID="71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140" Oct 29 21:54:04.885162 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:04.885135 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140\": container with ID starting with 71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140 not found: ID does not exist" containerID="71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140" Oct 29 21:54:04.885242 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.885180 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140"} err="failed to get container status \"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140\": rpc error: code = NotFound desc = could not find container \"71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140\": container with ID starting with 71e2356a4717b9d0f587e175fe3fcdfac9a3ba8cb63a9f2ef05c6c1ca0ca8140 not found: ID does not exist" Oct 29 21:54:04.885242 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.885208 4376 scope.go:122] "RemoveContainer" containerID="e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf" Oct 29 21:54:04.885504 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:04.885483 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf\": container with ID starting with e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf not found: ID does not exist" containerID="e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf" Oct 29 21:54:04.885566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.885510 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf"} err="failed to get container status \"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf\": rpc error: code = NotFound desc = could not find container \"e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf\": container with ID starting with e9d1e669890a85190d67dacd48164d8cbff796af502449c598a172d7278819cf not found: ID does not exist" Oct 29 21:54:04.966841 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.966768 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podStartSLOduration=1.9667470649999998 podStartE2EDuration="1.966747065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:04.925763869 +0000 UTC m=+703.774240762" watchObservedRunningTime="2026-10-29 21:54:04.966747065 +0000 UTC m=+703.815223914" Oct 29 21:54:04.968001 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.967972 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:54:04.974177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:04.974131 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2976cd0d-openshift-default-6dd847d6f-qgpmm"] Oct 29 21:54:05.056612 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.056574 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 21:54:05.057237 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.057215 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:05.057237 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.057235 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:05.057424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.057269 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:05.057424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.057276 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:05.057424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.057408 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:05.061031 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.061002 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.081383 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.081329 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-92d984-serving-cert\"" Oct 29 21:54:05.100775 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.100680 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 21:54:05.145482 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.145443 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.145643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.145499 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhxb\" (UniqueName: \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.145692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.145669 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.145768 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.145754 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.226630 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.226587 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:05.232633 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.232602 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:05.247279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.247247 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.247279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.247295 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhxb\" (UniqueName: \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.247541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.247353 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.247541 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.247399 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.248215 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.248184 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.249864 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.249844 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.249961 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.249882 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.271940 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.271887 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" path="/var/lib/kubelet/pods/b3b86e2b-8441-4ec1-b19b-b29b06a5f96a/volumes" Oct 29 21:54:05.273061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.273019 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhxb\" (UniqueName: \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\") pod \"maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.397525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.397433 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:05.703821 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.703794 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 21:54:05.705581 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:05.705555 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63ec822_782a_41ee_b6d0_81ae8a852fc3.slice/crio-1da9394e0ec333161142e99f77d87a1aaedd845ef81ec665e0e0f4684f0bd7cb WatchSource:0}: Error finding container 1da9394e0ec333161142e99f77d87a1aaedd845ef81ec665e0e0f4684f0bd7cb: Status 404 returned error can't find the container with id 1da9394e0ec333161142e99f77d87a1aaedd845ef81ec665e0e0f4684f0bd7cb Oct 29 21:54:05.799261 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.799231 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 21:54:05.873472 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.873414 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" event={"ID":"a63ec822-782a-41ee-b6d0-81ae8a852fc3","Type":"ContainerStarted","Data":"1da9394e0ec333161142e99f77d87a1aaedd845ef81ec665e0e0f4684f0bd7cb"} Oct 29 21:54:05.873824 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.873809 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:05.875151 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:05.875116 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:06.881313 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:06.881273 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" event={"ID":"a63ec822-782a-41ee-b6d0-81ae8a852fc3","Type":"ContainerStarted","Data":"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e"} Oct 29 21:54:06.929833 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:06.929750 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" podStartSLOduration=1.5752660120000002 podStartE2EDuration="1.929726058s" totalImagesPullingTime="354.460046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:06.926257233 +0000 UTC m=+705.774734080" watchObservedRunningTime="2026-10-29 21:54:06.929726058 +0000 UTC m=+705.778202908" Oct 29 21:54:07.887407 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:07.887377 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:13.901933 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:13.901883 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 21:54:16.468009 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:16.467959 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:54:17.460572 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.460525 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 21:54:17.461183 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.461164 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b86e2b-8441-4ec1-b19b-b29b06a5f96a" containerName="istio-proxy" Oct 29 21:54:17.465148 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.465115 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.480965 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.480241 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 21:54:17.481533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.481504 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-82db63-serving-cert\"" Oct 29 21:54:17.499552 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.499514 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.499759 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.499576 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lnb\" (UniqueName: \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.499759 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.499655 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.499895 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.499837 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.601098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.601051 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.601292 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.601114 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.601292 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.601168 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lnb\" (UniqueName: \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.601292 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:17.601216 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-serving-cert: secret "maas-api-e2e-worker-w3-82db63-serving-cert" not found Oct 29 21:54:17.601292 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.601229 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.601292 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:17.601295 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls podName:34b5ca49-5e59-48df-b0a7-838f91298c62 nodeName:}" failed. No retries permitted until 2026-10-29 21:54:18.101272825 +0000 UTC m=+716.949749649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls") pod "maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" (UID: "34b5ca49-5e59-48df-b0a7-838f91298c62") : secret "maas-api-e2e-worker-w3-82db63-serving-cert" not found Oct 29 21:54:17.601844 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.601817 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.603871 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.603848 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:17.650877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:17.650799 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lnb\" (UniqueName: \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:18.110962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:18.110826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:18.113680 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:18.113640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:18.384595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:18.384489 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:18.772847 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:18.772814 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 21:54:18.774032 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:18.773993 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b5ca49_5e59_48df_b0a7_838f91298c62.slice/crio-f3f2ca888680033b03efacdfc6cb6c7b2148d798865850a6b802d3371b6def5c WatchSource:0}: Error finding container f3f2ca888680033b03efacdfc6cb6c7b2148d798865850a6b802d3371b6def5c: Status 404 returned error can't find the container with id f3f2ca888680033b03efacdfc6cb6c7b2148d798865850a6b802d3371b6def5c Oct 29 21:54:18.989740 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:18.989688 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" event={"ID":"34b5ca49-5e59-48df-b0a7-838f91298c62","Type":"ContainerStarted","Data":"f3f2ca888680033b03efacdfc6cb6c7b2148d798865850a6b802d3371b6def5c"} Oct 29 21:54:19.459493 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.459431 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 21:54:19.464835 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.464795 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.491641 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.491591 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 21:54:19.491899 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.491881 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-cea94a-serving-cert\"" Oct 29 21:54:19.576821 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.576783 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.576821 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.576823 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.577178 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.577120 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.577267 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.577248 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vks\" (UniqueName: \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.678983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.678934 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vks\" (UniqueName: \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.679180 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.679023 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.679180 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.679053 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.679282 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.679248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.680723 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.680678 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.683859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.683776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.687559 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.687490 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.705648 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.705616 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vks\" (UniqueName: \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\") pod \"maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.797271 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.797178 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:19.998183 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.998135 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" event={"ID":"34b5ca49-5e59-48df-b0a7-838f91298c62","Type":"ContainerStarted","Data":"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59"} Oct 29 21:54:19.998394 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:19.998288 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:20.062974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:20.059888 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" podStartSLOduration=2.722069425 podStartE2EDuration="3.059868426s" totalImagesPullingTime="337.799001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:20.041409997 +0000 UTC m=+718.889886867" watchObservedRunningTime="2026-10-29 21:54:20.059868426 +0000 UTC m=+718.908345271" Oct 29 21:54:20.166595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:20.164523 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 21:54:20.170583 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:20.170505 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb09b2e_a108_4f20_8404_a3d06e01e1c4.slice/crio-c870c4d38558f62aaef7bf1d9bdaec28efc2cb186186256272ea73d2246ffd8e WatchSource:0}: Error finding container c870c4d38558f62aaef7bf1d9bdaec28efc2cb186186256272ea73d2246ffd8e: Status 404 returned error can't find the container with id c870c4d38558f62aaef7bf1d9bdaec28efc2cb186186256272ea73d2246ffd8e Oct 29 21:54:21.005616 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:21.005579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" event={"ID":"5fb09b2e-a108-4f20-8404-a3d06e01e1c4","Type":"ContainerStarted","Data":"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c"} Oct 29 21:54:21.006181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:21.005763 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:21.006181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:21.005799 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" event={"ID":"5fb09b2e-a108-4f20-8404-a3d06e01e1c4","Type":"ContainerStarted","Data":"c870c4d38558f62aaef7bf1d9bdaec28efc2cb186186256272ea73d2246ffd8e"} Oct 29 21:54:24.027509 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.027468 4376 generic.go:356] "Generic (PLEG): container finished" podID="17317361-6678-4683-9202-e15636063a6b" containerID="5b6b270d46863b583ed577d3f86f979613b612771a08acd6fa13632aa0ae45b9" exitCode=0 Oct 29 21:54:24.028163 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.027628 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerDied","Data":"5b6b270d46863b583ed577d3f86f979613b612771a08acd6fa13632aa0ae45b9"} Oct 29 21:54:24.028163 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.028055 4376 scope.go:122] "RemoveContainer" containerID="5b6b270d46863b583ed577d3f86f979613b612771a08acd6fa13632aa0ae45b9" Oct 29 21:54:24.030858 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.030814 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:24.031026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.030946 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:24.031026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.030998 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:24.099568 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:24.099505 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" podStartSLOduration=4.696386857 podStartE2EDuration="5.099488604s" totalImagesPullingTime="403.101747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:21.058316699 +0000 UTC m=+719.906793549" watchObservedRunningTime="2026-10-29 21:54:24.099488604 +0000 UTC m=+722.947965488" Oct 29 21:54:25.061533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:25.061491 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerStarted","Data":"8341632414bb22448207390f02512acf413b809ade6cb052d211209ed5963bf7"} Oct 29 21:54:25.304604 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:25.304562 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:54:25.306038 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:25.306010 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:25.306177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:25.306063 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:25.757929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:25.757047 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:54:26.011641 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:26.011565 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 21:54:26.304613 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:26.303964 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:26.304613 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:26.304052 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:27.017850 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:27.017819 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 21:54:27.304881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:27.304329 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:27.304881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:27.304404 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:28.304187 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:28.304145 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:28.304391 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:28.304231 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:29.304819 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:29.304786 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:29.306117 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:29.304842 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:30.101084 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.101045 4376 generic.go:356] "Generic (PLEG): container finished" podID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerID="8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562" exitCode=0 Oct 29 21:54:30.101300 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.101123 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerDied","Data":"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562"} Oct 29 21:54:30.101573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.101554 4376 scope.go:122] "RemoveContainer" containerID="8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562" Oct 29 21:54:30.102968 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.102936 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:30.103033 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.103009 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:30.103073 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.103044 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:30.306947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.304250 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 29 21:54:30.306947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:30.304308 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 29 21:54:31.108526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:31.108494 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerStarted","Data":"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8"} Oct 29 21:54:31.309842 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:31.309808 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:54:31.314272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:31.314245 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 21:54:32.048941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:32.048470 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:32.048941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:32.048616 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 21:54:32.048941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:32.048661 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 21:54:32.809057 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:32.809017 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:33.041361 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:33.041324 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 21:54:33.041635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:33.041602 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 21:54:34.057932 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:34.057315 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 21:54:34.057932 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:34.057378 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 21:54:35.042153 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:35.041649 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 21:54:35.042153 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:35.041711 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 21:54:36.053929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:36.040176 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 21:54:36.053929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:36.040241 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 21:54:36.915059 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:36.915022 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:54:36.915257 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:36.915086 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:54:37.045480 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:37.045446 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:37.046465 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:37.046439 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 21:54:38.181005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.180233 4376 generic.go:356] "Generic (PLEG): container finished" podID="2d26d931-118e-4738-9aaa-a0074161de15" containerID="db58965ebd58993603641e2682d9ab0691c5f758d4f072466b9875f81982b925" exitCode=0 Oct 29 21:54:38.181005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.180433 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerDied","Data":"db58965ebd58993603641e2682d9ab0691c5f758d4f072466b9875f81982b925"} Oct 29 21:54:38.181827 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.181794 4376 scope.go:122] "RemoveContainer" containerID="db58965ebd58993603641e2682d9ab0691c5f758d4f072466b9875f81982b925" Oct 29 21:54:38.193113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.193059 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:38.193425 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.193394 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:38.193593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:38.193574 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:39.188054 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:39.188016 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerStarted","Data":"e37c3bb82f4b0cb501eb410f5221a512473cc7b4a3adc3fa4d49800999471679"} Oct 29 21:54:39.226516 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:39.226470 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:39.226724 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:39.226700 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:39.226809 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:39.226753 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:40.226621 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:40.226589 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:40.227041 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:40.226654 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:41.226823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:41.226789 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:41.227272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:41.226839 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:42.226628 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:42.226597 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:42.226815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:42.226646 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:43.227061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:43.227026 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:43.227457 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:43.227092 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:44.226064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:44.226030 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 21:54:44.226308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:44.226081 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 21:54:45.230490 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:45.230460 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:45.231032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:45.231011 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:45.231628 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:45.231606 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 21:54:46.275499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.275465 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 21:54:46.279731 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.279706 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.314254 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.314218 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29c2f2-openshift-default-dockercfg-t47zd\"" Oct 29 21:54:46.331331 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.331292 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 21:54:46.357182 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357138 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357381 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357484 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357451 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd44p\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357548 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357535 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357606 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357560 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357627 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357773 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.357829 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.357793 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.434304 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:54:46.434261 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:54:46.458775 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458737 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.458775 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458793 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd44p\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458866 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458896 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.458963 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459000 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459071 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459100 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459469 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459387 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459469 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459404 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459617 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459598 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459655 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.459753 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.459729 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.461619 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.461589 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.461733 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.461667 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.483883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.483845 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.484232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.484206 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd44p\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.631474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.631372 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:46.963543 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.963512 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 21:54:46.966262 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:54:46.966233 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a71abd_5771_4c76_a767_54ff21dad866.slice/crio-06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f WatchSource:0}: Error finding container 06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f: Status 404 returned error can't find the container with id 06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f Oct 29 21:54:46.968881 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.968843 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:46.969026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.968969 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:46.969026 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:46.969004 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:54:47.239544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:47.239256 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" event={"ID":"15a71abd-5771-4c76-a767-54ff21dad866","Type":"ContainerStarted","Data":"8266e0323206698da35084cf2c36f297e916d2e5a10f2608a61a3e90eb3b9e00"} Oct 29 21:54:47.239544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:47.239312 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" event={"ID":"15a71abd-5771-4c76-a767-54ff21dad866","Type":"ContainerStarted","Data":"06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f"} Oct 29 21:54:47.307926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:47.306276 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podStartSLOduration=1.306254428 podStartE2EDuration="1.306254428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:47.302615117 +0000 UTC m=+746.151091965" watchObservedRunningTime="2026-10-29 21:54:47.306254428 +0000 UTC m=+746.154731274" Oct 29 21:54:47.631525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:47.631426 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:48.651929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:48.637052 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 29 21:54:48.651929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:48.637136 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 29 21:54:49.632776 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:49.632728 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 29 21:54:49.632984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:49.632820 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 29 21:54:50.632448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:50.632415 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 29 21:54:50.638944 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:50.632477 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 29 21:54:51.632426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:51.632396 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 29 21:54:51.632669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:51.632647 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 29 21:54:52.631984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:52.631943 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 29 21:54:52.632173 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:52.632000 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 29 21:54:53.738597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:53.738560 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:53.739114 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:53.738972 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:54:53.739939 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:54:53.739887 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 21:55:14.946414 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:14.946319 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 21:55:14.950479 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:14.950458 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:14.969823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:14.969796 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 21:55:15.029143 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.029102 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"kube-root-ca.crt\"" Oct 29 21:55:15.029143 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.029119 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"openshift-service-ca.crt\"" Oct 29 21:55:15.029143 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.029129 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"default-dockercfg-x7kjc\"" Oct 29 21:55:15.029380 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.029193 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"premium-w6-863f1b-kserve-self-signed-certs\"" Oct 29 21:55:15.077919 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.077880 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.078086 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.077948 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.078086 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.078061 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.078159 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.078120 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.078159 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.078145 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.078239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.078223 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjl55\" (UniqueName: \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179503 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179572 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179610 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjl55\" (UniqueName: \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179763 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179789 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.179827 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.180593 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.180301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.181153 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.180542 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.204981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.202590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.205883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.205846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.214460 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.214424 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.220862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.220837 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjl55\" (UniqueName: \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\") pod \"premium-w6-863f1b-kserve-79bc7f5bbd-9p28s\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.278889 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.278863 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:15.583575 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:15.583544 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 21:55:15.585139 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:55:15.585115 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e69a718_86ef_4ea7_a475_fc5e7fe62476.slice/crio-bcf86f1ec9067f912722a2532b14dc5835a69ed99446df25c4a0c44d2f9d1438 WatchSource:0}: Error finding container bcf86f1ec9067f912722a2532b14dc5835a69ed99446df25c4a0c44d2f9d1438: Status 404 returned error can't find the container with id bcf86f1ec9067f912722a2532b14dc5835a69ed99446df25c4a0c44d2f9d1438 Oct 29 21:55:16.439217 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:55:16.439176 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:55:16.441586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:16.441549 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" event={"ID":"4e69a718-86ef-4ea7-a475-fc5e7fe62476","Type":"ContainerStarted","Data":"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3"} Oct 29 21:55:16.441720 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:16.441600 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" event={"ID":"4e69a718-86ef-4ea7-a475-fc5e7fe62476","Type":"ContainerStarted","Data":"bcf86f1ec9067f912722a2532b14dc5835a69ed99446df25c4a0c44d2f9d1438"} Oct 29 21:55:16.493685 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:16.490806 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podStartSLOduration=2.490786895 podStartE2EDuration="2.490786895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:16.487660314 +0000 UTC m=+775.336137163" watchObservedRunningTime="2026-10-29 21:55:16.490786895 +0000 UTC m=+775.339263743" Oct 29 21:55:17.278977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:17.278951 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:25.279417 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:25.279382 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:25.295136 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:25.295102 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:25.504967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:25.504938 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:55:31.342568 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.342533 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 21:55:31.348363 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.348340 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.381380 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.381346 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-91520c-openshift-default-dockercfg-gg8xj\"" Oct 29 21:55:31.384248 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.384218 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 21:55:31.412994 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.412957 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413156 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413004 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvdq\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413156 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413085 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413275 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413218 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413310 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413339 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413369 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413466 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.413525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.413518 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.514751 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.514713 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.514987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.514774 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.514987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.514880 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvdq\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.514987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.514940 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515028 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515090 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515116 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515146 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515185 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515183 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515435 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515560 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515538 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515640 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515699 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515687 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.515892 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.515872 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.517423 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.517362 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.517786 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.517763 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.551949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.551915 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvdq\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.558348 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.558281 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.665192 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.665093 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:31.982263 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.982236 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 21:55:31.983157 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:55:31.983131 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a36fd4_d33d_4038_8436_ca2a12b7fa25.slice/crio-1a7220c80904c19fadb6f654773c127ae88f3af88056fcc210ea243251865078 WatchSource:0}: Error finding container 1a7220c80904c19fadb6f654773c127ae88f3af88056fcc210ea243251865078: Status 404 returned error can't find the container with id 1a7220c80904c19fadb6f654773c127ae88f3af88056fcc210ea243251865078 Oct 29 21:55:31.985766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.985731 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:55:31.985833 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.985802 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:55:31.985870 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:31.985837 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:55:32.536999 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:32.536965 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" event={"ID":"43a36fd4-d33d-4038-8436-ca2a12b7fa25","Type":"ContainerStarted","Data":"e8b15e48f160cdc34e35fc7d8dc1ee16717ff0b3b5c876f5d8dd9b1e14d4cc5d"} Oct 29 21:55:32.537395 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:32.537018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" event={"ID":"43a36fd4-d33d-4038-8436-ca2a12b7fa25","Type":"ContainerStarted","Data":"1a7220c80904c19fadb6f654773c127ae88f3af88056fcc210ea243251865078"} Oct 29 21:55:32.602136 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:32.602077 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podStartSLOduration=1.6020619680000001 podStartE2EDuration="1.602061968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:32.599138141 +0000 UTC m=+791.447614991" watchObservedRunningTime="2026-10-29 21:55:32.602061968 +0000 UTC m=+791.450538814" Oct 29 21:55:32.665697 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:32.665662 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:33.668012 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:33.667976 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:33.668399 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:33.668030 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:34.666583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:34.666539 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:34.666765 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:34.666601 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:35.666507 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:35.666470 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:35.667043 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:35.666532 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:36.666275 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:36.666234 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:36.666581 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:36.666297 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:37.670319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:37.670272 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:37.670846 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:37.670343 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:38.665914 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:38.665863 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 21:55:38.666090 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:38.665961 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 21:55:39.670071 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:39.670036 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:39.670452 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:39.670403 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:39.671057 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:55:39.671034 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 21:55:46.433217 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:55:46.433180 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:56:16.433732 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:56:16.433695 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:56:18.168985 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.168943 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 21:56:18.173170 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.173146 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.191551 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.191527 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82db63-openshift-default-dockercfg-7p225\"" Oct 29 21:56:18.202780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.202749 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.202973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.202789 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.202973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.202811 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203122 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203027 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203124 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjksh\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203181 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203172 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203289 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203314 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.203417 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.203380 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.209030 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.209001 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 21:56:18.304216 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304176 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304235 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304295 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304323 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304515 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304627 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304590 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304674 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjksh\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304679 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304726 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304735 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304789 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.304880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.304841 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.305061 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.305045 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.305105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.305090 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.306754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.306730 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.307138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.307119 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.331388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.331355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.331508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.331422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjksh\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.487389 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.487352 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:18.809830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.809803 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 21:56:18.811027 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:56:18.810988 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode798caae_4b39_4079_8849_1a4bbae067ca.slice/crio-f6cf3a3d1b39f177d2763bee0284a3f338cfe2bdb69913e4f7f7dabf67ad7a1d WatchSource:0}: Error finding container f6cf3a3d1b39f177d2763bee0284a3f338cfe2bdb69913e4f7f7dabf67ad7a1d: Status 404 returned error can't find the container with id f6cf3a3d1b39f177d2763bee0284a3f338cfe2bdb69913e4f7f7dabf67ad7a1d Oct 29 21:56:18.813375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.813342 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:18.813459 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.813409 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:18.813459 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.813439 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:18.889544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.889504 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" event={"ID":"e798caae-4b39-4079-8849-1a4bbae067ca","Type":"ContainerStarted","Data":"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d"} Oct 29 21:56:18.889733 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.889572 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" event={"ID":"e798caae-4b39-4079-8849-1a4bbae067ca","Type":"ContainerStarted","Data":"f6cf3a3d1b39f177d2763bee0284a3f338cfe2bdb69913e4f7f7dabf67ad7a1d"} Oct 29 21:56:18.948964 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:18.948895 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podStartSLOduration=0.948881131 podStartE2EDuration="948.881131ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:18.946771436 +0000 UTC m=+837.795248282" watchObservedRunningTime="2026-10-29 21:56:18.948881131 +0000 UTC m=+837.797358029" Oct 29 21:56:19.488252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:19.488215 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:19.490032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:19.490003 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:19.490141 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:19.490061 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:20.488388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:20.488345 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:20.488821 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:20.488402 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:21.488785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:21.488747 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:21.489329 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:21.488809 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:22.488390 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:22.488350 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:22.488575 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:22.488418 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:23.488018 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:23.487983 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:23.488426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:23.488054 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:24.493391 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:24.493353 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 21:56:24.494751 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:24.494716 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 21:56:25.492858 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:25.492827 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:25.493142 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:25.493125 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:25.493868 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:25.493848 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 21:56:31.616341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.616301 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 21:56:31.620721 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.620695 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.655205 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.655170 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 21:56:31.728023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.727985 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728032 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728298 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728049 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728298 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728298 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728298 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728288 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728511 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728341 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728511 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728409 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.728511 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.728448 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829555 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829522 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829570 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829594 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829626 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829674 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829713 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829736 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.829767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829761 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.829829 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830210 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.830188 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830263 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.830218 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.830305 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.830409 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.830666 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.830652 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.831999 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.831979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.832222 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.832203 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.856246 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.856216 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.856386 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.856267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:31.935431 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:31.935340 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:32.252169 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:32.252135 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 21:56:32.253437 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:56:32.253413 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c04004_8e34_43fd_bce0_cf58df2383f4.slice/crio-af66117f3878f1955698d18cfe06e4de611745d7ba83e103155ff726c6f329cc WatchSource:0}: Error finding container af66117f3878f1955698d18cfe06e4de611745d7ba83e103155ff726c6f329cc: Status 404 returned error can't find the container with id af66117f3878f1955698d18cfe06e4de611745d7ba83e103155ff726c6f329cc Oct 29 21:56:32.255874 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:32.255835 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:32.255952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:32.255922 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:32.255999 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:32.255952 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:56:33.062843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.062811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" event={"ID":"f6c04004-8e34-43fd-bce0-cf58df2383f4","Type":"ContainerStarted","Data":"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31"} Oct 29 21:56:33.063260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.062859 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" event={"ID":"f6c04004-8e34-43fd-bce0-cf58df2383f4","Type":"ContainerStarted","Data":"af66117f3878f1955698d18cfe06e4de611745d7ba83e103155ff726c6f329cc"} Oct 29 21:56:33.127858 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.127798 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podStartSLOduration=2.127784601 podStartE2EDuration="2.127784601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:33.125334014 +0000 UTC m=+851.973810860" watchObservedRunningTime="2026-10-29 21:56:33.127784601 +0000 UTC m=+851.976261446" Oct 29 21:56:33.953135 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.943282 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:33.958097 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.958056 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 21:56:33.958265 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:33.958122 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 21:56:34.953928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:34.946288 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 21:56:34.953928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:34.946365 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 21:56:35.945396 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:35.945354 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 21:56:35.945678 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:35.945650 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 21:56:36.950930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:36.945358 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 21:56:36.950930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:36.945425 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 21:56:37.942226 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:37.942186 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 21:56:37.942532 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:37.942500 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 21:56:39.104074 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:39.104025 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:39.104754 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:39.104714 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:39.105850 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:39.105825 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 21:56:46.436664 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:56:46.436618 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:56:47.454147 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.454111 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 21:56:47.459452 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.459329 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.475342 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.475312 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 21:56:47.476099 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476073 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.476243 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476222 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.476313 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476294 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.476427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476405 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.476483 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476450 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmj8\" (UniqueName: \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.476573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.476544 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.485374 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.485352 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"distinct-w6-863f1b-kserve-self-signed-certs\"" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577507 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577576 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577614 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmj8\" (UniqueName: \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577648 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577802 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.577888 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.578360 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.578773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.578862 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.578818 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.581470 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.581441 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.581827 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.581800 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.604346 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.604318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmj8\" (UniqueName: \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\") pod \"distinct-w6-863f1b-kserve-8676d45f59-9gqm7\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:47.781201 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:47.781110 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:48.086859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:48.086826 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 21:56:48.087791 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:56:48.087763 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c364880_627c_428e_8713_6d29733874bd.slice/crio-df1c7fbf0a5ac28ca2ad249e1e648644c19473366310a583bd31bb589c74289a WatchSource:0}: Error finding container df1c7fbf0a5ac28ca2ad249e1e648644c19473366310a583bd31bb589c74289a: Status 404 returned error can't find the container with id df1c7fbf0a5ac28ca2ad249e1e648644c19473366310a583bd31bb589c74289a Oct 29 21:56:48.168508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:48.168461 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" event={"ID":"7c364880-627c-428e-8713-6d29733874bd","Type":"ContainerStarted","Data":"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76"} Oct 29 21:56:48.168664 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:48.168523 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" event={"ID":"7c364880-627c-428e-8713-6d29733874bd","Type":"ContainerStarted","Data":"df1c7fbf0a5ac28ca2ad249e1e648644c19473366310a583bd31bb589c74289a"} Oct 29 21:56:48.215578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:48.215529 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podStartSLOduration=1.215514631 podStartE2EDuration="1.215514631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:48.214364432 +0000 UTC m=+867.062841290" watchObservedRunningTime="2026-10-29 21:56:48.215514631 +0000 UTC m=+867.063991521" Oct 29 21:56:48.782290 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:48.782259 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:52.878166 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.878130 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 21:56:52.883295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.883267 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.905419 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.905384 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 21:56:52.915395 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.915359 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"distinct-w3-82db63-kserve-self-signed-certs\"" Oct 29 21:56:52.915614 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.915498 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"kube-root-ca.crt\"" Oct 29 21:56:52.915810 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.915784 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"openshift-service-ca.crt\"" Oct 29 21:56:52.915966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.915882 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"default-dockercfg-dptlb\"" Oct 29 21:56:52.948007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.947976 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.948175 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.948028 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.948219 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.948161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.948219 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.948193 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p6j\" (UniqueName: \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.948327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.948314 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:52.948365 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:52.948348 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050005 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.049968 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050186 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050024 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050186 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050082 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050186 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050101 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p6j\" (UniqueName: \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050190 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050295 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050231 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050525 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050501 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050502 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.050595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.050551 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.052474 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.052446 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.052616 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.052598 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.105358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.105263 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p6j\" (UniqueName: \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\") pod \"distinct-w3-82db63-kserve-85fdcb446f-vxxcw\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.206228 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.206199 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:53.509167 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:53.509143 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 21:56:53.510869 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:56:53.510839 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19e4df8_c3f6_4acf_95b1_62dfcb501bf3.slice/crio-9586b5400664d4a777c3270535a364ddd53d6c7844a36b6ab603b386becc8b14 WatchSource:0}: Error finding container 9586b5400664d4a777c3270535a364ddd53d6c7844a36b6ab603b386becc8b14: Status 404 returned error can't find the container with id 9586b5400664d4a777c3270535a364ddd53d6c7844a36b6ab603b386becc8b14 Oct 29 21:56:54.204021 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:54.203979 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" event={"ID":"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3","Type":"ContainerStarted","Data":"2d6203ae1fe152e0cb9aa0db23797fdff065af4531b548bd5187422f398ffbf5"} Oct 29 21:56:54.204430 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:54.204033 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" event={"ID":"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3","Type":"ContainerStarted","Data":"9586b5400664d4a777c3270535a364ddd53d6c7844a36b6ab603b386becc8b14"} Oct 29 21:56:54.206304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:54.206282 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:56:54.251484 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:54.251415 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podStartSLOduration=2.251396729 podStartE2EDuration="2.251396729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:54.249511181 +0000 UTC m=+873.097988026" watchObservedRunningTime="2026-10-29 21:56:54.251396729 +0000 UTC m=+873.099873578" Oct 29 21:56:57.781736 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:57.781700 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:57.794363 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:57.794337 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:56:58.291172 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:56:58.291144 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:57:01.602154 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.602111 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 21:57:01.607281 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.607252 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.626216 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.626182 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 21:57:01.635801 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.635768 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"distinct-2-w1-29c2f2-kserve-self-signed-certs\"" Oct 29 21:57:01.636000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.635804 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"default-dockercfg-58jq5\"" Oct 29 21:57:01.638364 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.638348 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"kube-root-ca.crt\"" Oct 29 21:57:01.638433 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.638408 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"openshift-service-ca.crt\"" Oct 29 21:57:01.714548 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714509 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.714737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714558 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.714737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714691 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.714882 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714862 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.714949 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714899 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.715046 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.714990 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whtd\" (UniqueName: \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.831930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832002 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832050 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whtd\" (UniqueName: \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832153 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832213 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.832827 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.845927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.843608 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.861627 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.850363 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.902526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.900661 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.902526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.901031 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.902768 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.902531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whtd\" (UniqueName: \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\") pod \"distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:01.921538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:01.921486 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:02.223130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:02.223101 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 21:57:02.224564 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:57:02.224524 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4698b55f_414e_42d3_a92f_ad7ecfeec77e.slice/crio-8dc01474b06f2fdd8298909d931bf21579bf23ecfdb6a731f547877656ab9c89 WatchSource:0}: Error finding container 8dc01474b06f2fdd8298909d931bf21579bf23ecfdb6a731f547877656ab9c89: Status 404 returned error can't find the container with id 8dc01474b06f2fdd8298909d931bf21579bf23ecfdb6a731f547877656ab9c89 Oct 29 21:57:02.307822 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:02.307784 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" event={"ID":"4698b55f-414e-42d3-a92f-ad7ecfeec77e","Type":"ContainerStarted","Data":"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f"} Oct 29 21:57:02.308007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:02.307836 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" event={"ID":"4698b55f-414e-42d3-a92f-ad7ecfeec77e","Type":"ContainerStarted","Data":"8dc01474b06f2fdd8298909d931bf21579bf23ecfdb6a731f547877656ab9c89"} Oct 29 21:57:02.358446 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:02.358357 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podStartSLOduration=1.358337534 podStartE2EDuration="1.358337534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:02.352362601 +0000 UTC m=+881.200839710" watchObservedRunningTime="2026-10-29 21:57:02.358337534 +0000 UTC m=+881.206814380" Oct 29 21:57:02.921885 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:02.921842 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:03.207327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:03.207293 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:57:03.220573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:03.220535 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:57:03.342567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:03.342532 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:57:11.923003 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:11.922965 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:12.025544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:12.025514 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:12.416651 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:12.416621 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:57:16.436592 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:57:16.436555 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:57:44.058231 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.058187 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 21:57:44.062883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.062851 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.084301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.084269 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 21:57:44.093171 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.093145 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"kube-root-ca.crt\"" Oct 29 21:57:44.093701 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.093677 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"openshift-service-ca.crt\"" Oct 29 21:57:44.093861 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.093734 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"distinct-2-w4-cea94a-kserve-self-signed-certs\"" Oct 29 21:57:44.093861 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.093745 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"default-dockercfg-s86jj\"" Oct 29 21:57:44.126537 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126505 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2z6\" (UniqueName: \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.126683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126553 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.126683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.126756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126686 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.126756 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126730 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.126880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.126868 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.227697 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227652 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.227922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2z6\" (UniqueName: \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.227922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227764 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.227922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227832 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.227922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227857 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.228163 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.227934 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.228163 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.228093 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.228303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.228270 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.228434 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.228348 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.230478 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.230455 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.230991 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.230972 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.296807 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.296772 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2z6\" (UniqueName: \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\") pod \"distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.379305 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.379200 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:44.695063 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:44.695030 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 21:57:44.695977 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:57:44.695946 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90f0d211_5adf_40ca_8c85_56878719bf15.slice/crio-dbf7ae115a61149252bc04fb1a264d30861dd0ec59ca6e77fde42caa62a8224e WatchSource:0}: Error finding container dbf7ae115a61149252bc04fb1a264d30861dd0ec59ca6e77fde42caa62a8224e: Status 404 returned error can't find the container with id dbf7ae115a61149252bc04fb1a264d30861dd0ec59ca6e77fde42caa62a8224e Oct 29 21:57:45.610920 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:45.610866 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" event={"ID":"90f0d211-5adf-40ca-8c85-56878719bf15","Type":"ContainerStarted","Data":"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a"} Oct 29 21:57:45.611311 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:45.610950 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" event={"ID":"90f0d211-5adf-40ca-8c85-56878719bf15","Type":"ContainerStarted","Data":"dbf7ae115a61149252bc04fb1a264d30861dd0ec59ca6e77fde42caa62a8224e"} Oct 29 21:57:45.657293 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:45.657233 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podStartSLOduration=1.657217945 podStartE2EDuration="1.657217945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:45.656653656 +0000 UTC m=+924.505130502" watchObservedRunningTime="2026-10-29 21:57:45.657217945 +0000 UTC m=+924.505694791" Oct 29 21:57:46.380126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:46.380090 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:46.484567 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:57:46.484533 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:57:52.511367 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.511334 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 21:57:52.515845 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.515822 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.554562 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.554532 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"kube-root-ca.crt\"" Oct 29 21:57:52.554731 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.554593 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"openshift-service-ca.crt\"" Oct 29 21:57:52.554731 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.554632 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"default-dockercfg-xqpdv\"" Oct 29 21:57:52.554731 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.554724 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"distinct-w0-f3b067-kserve-self-signed-certs\"" Oct 29 21:57:52.559301 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.559271 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 21:57:52.679741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679704 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.679741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679741 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.679981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679769 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8sx\" (UniqueName: \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.679981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679881 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.679981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679973 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.680100 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.679994 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790080 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790249 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8sx\" (UniqueName: \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790348 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790415 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.790438 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.791886 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.792440 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.797972 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.792686 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.810924 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.809539 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.822723 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.822690 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8sx\" (UniqueName: \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.829321 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.825418 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\") pod \"distinct-w0-f3b067-kserve-56b74fb959-cfhck\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:52.832174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:52.832144 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:53.165601 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:53.165574 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 21:57:53.166947 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:57:53.166892 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee972f81_b171_40b5_8642_a8f34172e90d.slice/crio-b6decea1c7964661de51ab0b4e12686c86ce9ee2d7e21d47eba87f93f9e62593 WatchSource:0}: Error finding container b6decea1c7964661de51ab0b4e12686c86ce9ee2d7e21d47eba87f93f9e62593: Status 404 returned error can't find the container with id b6decea1c7964661de51ab0b4e12686c86ce9ee2d7e21d47eba87f93f9e62593 Oct 29 21:57:53.655888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:53.655851 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" event={"ID":"ee972f81-b171-40b5-8642-a8f34172e90d","Type":"ContainerStarted","Data":"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a"} Oct 29 21:57:53.655888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:53.655894 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" event={"ID":"ee972f81-b171-40b5-8642-a8f34172e90d","Type":"ContainerStarted","Data":"b6decea1c7964661de51ab0b4e12686c86ce9ee2d7e21d47eba87f93f9e62593"} Oct 29 21:57:53.702491 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:53.702439 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podStartSLOduration=1.702425323 podStartE2EDuration="1.702425323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:53.700735731 +0000 UTC m=+932.549212579" watchObservedRunningTime="2026-10-29 21:57:53.702425323 +0000 UTC m=+932.550902186" Oct 29 21:57:53.846926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:53.846493 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:57:54.379917 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.379871 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:54.399830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.399790 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:54.680938 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.680521 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:57:54.690099 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.690059 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 21:57:54.695024 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.694998 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.733641 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.733608 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 21:57:54.830179 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830140 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830342 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830304 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830533 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830510 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830557 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jr4\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830815 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830852 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.830973 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.830892 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.831062 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.831034 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.831165 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.831147 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943022 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943089 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jr4\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943206 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943364 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943400 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.951979 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.943459 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.968114 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.968080 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.970616 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.970590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.973811 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.973758 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.974476 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.974449 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:54.974764 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:54.974740 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.004546 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.004465 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.012995 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.011323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.012995 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.011879 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.024668 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.024601 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jr4\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.042036 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.029722 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:55.620880 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.620827 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:57:55.621177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.621140 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:57:55.621388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.621359 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:57:55.754439 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.754401 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 21:57:55.761097 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:55.761064 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" event={"ID":"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f","Type":"ContainerStarted","Data":"785f50b6124092ad0e26d002974eb3ea585299f39381429b8f99f3e1299534cd"} Oct 29 21:57:56.790140 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:56.790100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" event={"ID":"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f","Type":"ContainerStarted","Data":"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22"} Oct 29 21:57:56.900898 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:56.900820 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podStartSLOduration=2.900800856 podStartE2EDuration="2.900800856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:56.855575986 +0000 UTC m=+935.704052833" watchObservedRunningTime="2026-10-29 21:57:56.900800856 +0000 UTC m=+935.749277703" Oct 29 21:57:57.030887 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:57.030811 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:57:57.032089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:57.032062 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:57:57.032198 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:57.032109 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:57:58.031220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:58.031181 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:57:58.031643 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:58.031254 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:57:59.031368 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:59.031331 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:57:59.031728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:57:59.031384 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:58:00.030462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:00.030424 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:58:00.030664 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:00.030493 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:58:01.031281 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:01.031238 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:58:01.031753 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:01.031309 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:58:02.030680 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:02.030646 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:58:02.030843 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:02.030709 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:58:02.835495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:02.835462 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:02.872511 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:02.872477 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:03.030566 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:03.030530 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:58:03.030732 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:03.030610 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 21:58:03.850499 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:03.850408 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:04.144777 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:04.144677 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:58:04.145201 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:04.145176 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:58:04.147382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:04.147357 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 21:58:16.433018 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:58:16.432984 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:58:24.222038 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.221989 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.283073 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.283023 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.323050 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.323004 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.674222 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.674116 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.788358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.788319 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:24.788687 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.788635 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.867517 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:24.867472 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.163512 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.157010 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:25.163512 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.157088 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.349044 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.348992 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.418010 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.377992 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.418010 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.382041 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.516526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.514605 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:25.601433 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.601401 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:58:25.605780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.603969 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:58:25.605780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.604032 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:58:25.675103 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.674983 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.719948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.705385 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:58:26.006428 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.908332 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.006428 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:25.941002 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:26.411118 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.411025 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:26.413254 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.413216 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.649084 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.649030 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.651091 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.651067 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:58:26.676221 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.676143 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 21:58:26.945854 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:26.945757 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:27.075922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:27.054028 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:27.075922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:27.064069 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:58:27.082755 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:27.079853 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:58:27.084180 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:27.083089 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:58:36.249227 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.249190 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 21:58:36.256272 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.256244 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.271722 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.271695 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 21:58:36.282433 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.282352 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"distinct-2-w0-f3b067-kserve-self-signed-certs\"" Oct 29 21:58:36.387174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387135 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvkq\" (UniqueName: \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.387339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387279 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.387424 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387409 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.387492 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387442 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.387540 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387506 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.387680 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.387659 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493003 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493180 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvkq\" (UniqueName: \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493212 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493317 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493361 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.493387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.494645 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.494214 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.496952 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.496871 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.497253 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.497225 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.518595 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.518516 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.525934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.523725 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.556667 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.556633 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvkq\" (UniqueName: \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\") pod \"distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.571221 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.571189 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:36.872877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:36.872845 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 21:58:36.874291 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:58:36.874265 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2a90d29_40a0_4aa8_9991_61f234b3e273.slice/crio-4643ef0ce7e572c923f6e0f7c2911e3d9c6b771fab23cf6b3d9d12493487148a WatchSource:0}: Error finding container 4643ef0ce7e572c923f6e0f7c2911e3d9c6b771fab23cf6b3d9d12493487148a: Status 404 returned error can't find the container with id 4643ef0ce7e572c923f6e0f7c2911e3d9c6b771fab23cf6b3d9d12493487148a Oct 29 21:58:37.758652 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:37.758621 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" event={"ID":"f2a90d29-40a0-4aa8-9991-61f234b3e273","Type":"ContainerStarted","Data":"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b"} Oct 29 21:58:37.759058 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:37.758666 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" event={"ID":"f2a90d29-40a0-4aa8-9991-61f234b3e273","Type":"ContainerStarted","Data":"4643ef0ce7e572c923f6e0f7c2911e3d9c6b771fab23cf6b3d9d12493487148a"} Oct 29 21:58:37.805382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:37.805334 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podStartSLOduration=1.805320176 podStartE2EDuration="1.805320176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:37.804693869 +0000 UTC m=+976.653170716" watchObservedRunningTime="2026-10-29 21:58:37.805320176 +0000 UTC m=+976.653797022" Oct 29 21:58:38.571612 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.571579 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:38.721925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.721874 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 21:58:38.725875 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.725856 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.769730 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.769696 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.820716 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.820849 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821039 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821082 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821112 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821184 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvfd\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821235 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.821766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.821291 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922509 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922555 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922579 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922601 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922621 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922682 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvfd\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922708 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922852 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.922976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922946 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.923116 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.922979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.923311 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.923280 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.923435 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.923384 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.923669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.923646 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.925225 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.925202 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.925326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.925260 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.948461 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.948433 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:38.948563 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:38.948545 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvfd\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:39.047737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.046234 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:39.400666 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.400643 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 21:58:39.402309 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:58:39.402278 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01627787_0bc7_467e_90c3_2e847b1fa504.slice/crio-b264f37df41ab57ac4220eb859cbb34a9ad9c35dde7eb73ba5805229348d5244 WatchSource:0}: Error finding container b264f37df41ab57ac4220eb859cbb34a9ad9c35dde7eb73ba5805229348d5244: Status 404 returned error can't find the container with id b264f37df41ab57ac4220eb859cbb34a9ad9c35dde7eb73ba5805229348d5244 Oct 29 21:58:39.405112 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.405079 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:39.405174 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.405144 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:39.405207 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.405178 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:39.772987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.772950 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" event={"ID":"01627787-0bc7-467e-90c3-2e847b1fa504","Type":"ContainerStarted","Data":"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db"} Oct 29 21:58:39.773374 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.773012 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" event={"ID":"01627787-0bc7-467e-90c3-2e847b1fa504","Type":"ContainerStarted","Data":"b264f37df41ab57ac4220eb859cbb34a9ad9c35dde7eb73ba5805229348d5244"} Oct 29 21:58:39.836480 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:39.836425 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podStartSLOduration=1.8364111520000002 podStartE2EDuration="1.836411152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:39.834935799 +0000 UTC m=+978.683412647" watchObservedRunningTime="2026-10-29 21:58:39.836411152 +0000 UTC m=+978.684887998" Oct 29 21:58:40.047197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:40.047124 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:40.048553 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:40.048526 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:40.048684 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:40.048582 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:41.049802 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:41.049331 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:41.049802 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:41.049394 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:42.057931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:42.049288 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:42.057931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:42.049351 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:43.058032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:43.057994 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:43.058440 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:43.058045 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:44.057930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:44.056248 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:44.057930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:44.056342 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:45.054926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:45.054332 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:45.054926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:45.054426 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:46.048132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:46.048095 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:58:46.048569 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:46.048170 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 21:58:46.438097 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:58:46.437991 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:58:46.572034 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:46.572000 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:46.585009 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:46.584979 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:46.838731 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:46.838700 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:47.033040 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.032996 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 21:58:47.038040 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.037980 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.052475 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.052444 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:47.054800 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.054776 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:47.056778 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.056759 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 21:58:47.069138 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.069111 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 21:58:47.082466 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082433 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.082602 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082492 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.082669 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082588 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.082773 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082755 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.082829 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082812 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57gwl\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.082943 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.082847 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.083203 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.083182 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.083340 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.083231 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.083340 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.083299 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184507 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184770 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184604 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184770 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184770 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184754 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184789 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184885 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.184967 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184959 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gwl\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.185121 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.184996 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.185121 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.185104 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192355 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192176 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192355 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192303 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192372 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192421 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192506 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192539 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.192792 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.192590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.216528 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.216504 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.216657 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.216581 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gwl\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.355297 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.355263 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:47.674741 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.674710 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 21:58:47.676795 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:58:47.676764 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d57a57_ac8c_43f8_a235_5b8952fba093.slice/crio-b557d0cb4a5cdd55aa3215ea0f0e7019b44cabb5204e6e2c70d0b74de101a06b WatchSource:0}: Error finding container b557d0cb4a5cdd55aa3215ea0f0e7019b44cabb5204e6e2c70d0b74de101a06b: Status 404 returned error can't find the container with id b557d0cb4a5cdd55aa3215ea0f0e7019b44cabb5204e6e2c70d0b74de101a06b Oct 29 21:58:47.679692 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.679627 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:47.679785 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.679727 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:47.679832 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.679784 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:47.955982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.951996 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" event={"ID":"f0d57a57-ac8c-43f8-a235-5b8952fba093","Type":"ContainerStarted","Data":"54d1d90d7b181bedf87e78ac216c4410d099e13a0f7fe18b2d47ed2dec88f8ed"} Oct 29 21:58:47.955982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:47.952195 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" event={"ID":"f0d57a57-ac8c-43f8-a235-5b8952fba093","Type":"ContainerStarted","Data":"b557d0cb4a5cdd55aa3215ea0f0e7019b44cabb5204e6e2c70d0b74de101a06b"} Oct 29 21:58:48.015239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:48.015161 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podStartSLOduration=1.015139378 podStartE2EDuration="1.015139378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:48.012375525 +0000 UTC m=+986.860852363" watchObservedRunningTime="2026-10-29 21:58:48.015139378 +0000 UTC m=+986.863616226" Oct 29 21:58:48.367349 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:48.366233 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:58:48.383927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:48.375117 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:48.383927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:48.375213 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:49.360239 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.360203 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:49.360568 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.360513 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:49.687106 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.687005 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:49.922928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.815962 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 21:58:49.981009 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.925987 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.021981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:49.981378 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.074962 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.057262 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 21:58:50.346987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.331106 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.347477 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.347428 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:50.348406 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.348364 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:50.434934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.432028 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.439929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.437494 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:50.439929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.437551 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457096 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457158 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457247 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb8s\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457362 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457402 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457444 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457582 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.483990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.457796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.560481 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.560436 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.567218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.561734 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.567436 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.567287 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.597977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.567625 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.598250 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.598222 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.598501 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.598478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb8s\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.598766 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.598731 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.599015 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.598994 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.599220 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.599201 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.599420 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.599402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.600136 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.600109 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.637002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.600555 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.637002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.594677 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.637002 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.600823 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.771927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.722220 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 21:58:50.771927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.723130 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:58:50.800926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.776400 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.819927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.809079 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.871734 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.821219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.872065 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.842779 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb8s\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:50.991930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.966007 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:51.030928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.992230 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 21:58:51.030928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:50.992680 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:51.138292 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.138190 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:51.420976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.395821 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:51.474929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.448679 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.535978 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.457986 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.535978 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.459169 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:51.584937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.580460 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:51.584937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.580523 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:51.748928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.747221 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.832558 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.832520 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:58:51.833175 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.833152 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:58:51.833350 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.833336 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:58:51.833494 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.833481 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:58:51.969108 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.969050 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.974698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:51.974668 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:52.011408 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.011317 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.428442 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.428338 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:52.524297 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.521885 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.539358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.533646 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:52.539358 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.534735 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.605439 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.605086 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:52.605439 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.605141 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:52.748795 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.748742 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.956562 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.956460 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.998326 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:52.998286 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:53.046986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.038493 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.394260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.394135 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:53.394260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.394192 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:53.447162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.445500 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:53.469631 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.469588 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:53.474038 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.474004 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:53.474238 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.474215 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:53.575547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.565897 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.575547 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.566445 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:53.630339 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.522358 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.753073 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.753025 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.840060 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.840016 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.859926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.848823 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 21:58:53.962285 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.961258 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:58:53.987363 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:53.987331 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" event={"ID":"fa6d09c5-b567-4547-b8e3-075959fccbe0","Type":"ContainerStarted","Data":"7d21a229c4b533a381693d7619304326d990766b2bf14e25868cdc6c1636254b"} Oct 29 21:58:54.013685 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.012725 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:54.232101 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.232049 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.396938 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.395333 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:54.396938 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.395395 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:54.479992 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.479072 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.576898 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.576850 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.577505 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.577473 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 21:58:54.646350 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.646310 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 21:58:54.947951 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.931788 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.993285 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:54.993240 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:58:55.114984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.107250 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.200697 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.200609 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 21:58:55.208598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.208576 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 21:58:55.360073 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.360040 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:55.360443 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.360315 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:55.530222 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.530093 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.613790 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.613739 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.681561 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.681535 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 21:58:55.681774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.681753 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:58:55.681859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.681795 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:58:55.681859 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.681814 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" event={"ID":"fa6d09c5-b567-4547-b8e3-075959fccbe0","Type":"ContainerStarted","Data":"207b6d44919378a4018d87a5fdb68bebab80696c7e7a42badc15e41d482005fa"} Oct 29 21:58:55.687252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:55.687156 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podStartSLOduration=6.687142853 podStartE2EDuration="6.687142853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:55.683980896 +0000 UTC m=+994.532457768" watchObservedRunningTime="2026-10-29 21:58:55.687142853 +0000 UTC m=+994.535619699" Oct 29 21:58:56.164930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:56.163999 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:58:56.165888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:56.165035 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.165888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:56.165090 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:58:56.367929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:56.365290 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.367929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:56.365351 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:57.110375 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.110327 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 21:58:57.154746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.154675 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.154746 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.154741 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:58:57.251244 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.236743 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 21:58:57.289772 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.238105 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.410324 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.410245 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a116a2-openshift-default-dockercfg-xwhg9\"" Oct 29 21:58:57.416303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.412808 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.416563 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.416538 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:57.417051 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.413595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.417319 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.417296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnrq\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.417479 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.417462 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.417602 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.417583 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.417883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.417865 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.418056 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.418037 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.418191 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.418167 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.418332 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.418311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.418465 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.418449 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.520931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.520871 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.522268 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.522240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.522498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.522478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.522658 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.522633 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.522809 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.522792 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.523238 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.523216 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.523567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.523545 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnrq\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.523718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.523703 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.523826 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.523813 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.524382 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.524361 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.524504 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.522185 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.524827 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.524809 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.525932 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.525890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.528500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.528214 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.568888 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.564964 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.584714 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.584681 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.603937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.602600 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.631938 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.630624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnrq\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\") pod \"e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:57.647197 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:57.634502 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:58.154928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.147396 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.154928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.147457 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:58:58.211190 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.211157 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 21:58:58.212645 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:58:58.212613 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81945ba_078c_49ff_a4ef_60ec6ed2e21e.slice/crio-3f80a2a3f4ddbcb45d1fa91c4dbdcbaaed67bbc79f89cd8a7a2d5844abb5ed50 WatchSource:0}: Error finding container 3f80a2a3f4ddbcb45d1fa91c4dbdcbaaed67bbc79f89cd8a7a2d5844abb5ed50: Status 404 returned error can't find the container with id 3f80a2a3f4ddbcb45d1fa91c4dbdcbaaed67bbc79f89cd8a7a2d5844abb5ed50 Oct 29 21:58:58.215003 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.214947 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:58.215071 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.215043 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:58.215126 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.215086 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:58:58.360925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.356205 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.360925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.356257 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 21:58:58.474245 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.474154 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerStarted","Data":"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba"} Oct 29 21:58:58.476136 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.476108 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerStarted","Data":"3f80a2a3f4ddbcb45d1fa91c4dbdcbaaed67bbc79f89cd8a7a2d5844abb5ed50"} Oct 29 21:58:58.528407 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.528354 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podStartSLOduration=2.528341196 podStartE2EDuration="2.528341196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:58.525793164 +0000 UTC m=+997.374270012" watchObservedRunningTime="2026-10-29 21:58:58.528341196 +0000 UTC m=+997.376818042" Oct 29 21:58:58.649038 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:58.648996 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:58:59.146928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:59.146334 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:58:59.146928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:59.146414 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:58:59.665937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:58:59.664289 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:59:00.153928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.148241 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:59:00.153928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.148302 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:59:00.411090 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.411008 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:00.411264 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.411129 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:00.487649 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.487616 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:59:00.488896 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:00.488874 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 21:59:01.094543 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:01.094513 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:59:01.095737 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:01.095710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:59:01.128703 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:01.128669 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 21:59:01.162937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:01.162309 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:59:01.162937 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:01.162377 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:59:02.148935 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:02.148258 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:59:02.148935 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:02.148332 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:59:03.194960 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:03.191186 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:59:03.194960 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:03.191249 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 21:59:04.149427 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:04.149397 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:59:04.149702 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:04.149680 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:59:04.150508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:04.150488 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 21:59:14.401760 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.401718 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 21:59:14.406568 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.406536 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.427397 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.427371 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8867e7d-openshift-default-dockercfg-s5lcp\"" Oct 29 21:59:14.441303 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.441275 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 21:59:14.500420 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500388 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500584 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500459 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500584 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500544 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmxn\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500584 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500579 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500751 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500682 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500805 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500781 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500869 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500802 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500946 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.500946 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.500930 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.602986 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603229 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603265 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmxn\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603294 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603725 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.603777 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.604773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.608355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.608629 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.608925 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.616387 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.617948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.616612 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.625469 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.618008 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.677974 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.664058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmxn\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.714352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.714287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:14.726434 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:14.726156 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:15.118291 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.118261 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 21:59:15.118884 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:15.118858 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd67725_79b9_499b_a238_61d27d1db363.slice/crio-0f671fd6a820dd96d7a96a9059e80dfbe68ccc7748614b4027b9484bb39dc2ec WatchSource:0}: Error finding container 0f671fd6a820dd96d7a96a9059e80dfbe68ccc7748614b4027b9484bb39dc2ec: Status 404 returned error can't find the container with id 0f671fd6a820dd96d7a96a9059e80dfbe68ccc7748614b4027b9484bb39dc2ec Oct 29 21:59:15.121365 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.121331 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:15.121448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.121398 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:15.121448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.121427 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:15.630639 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.630602 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerStarted","Data":"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23"} Oct 29 21:59:15.631048 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.630658 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerStarted","Data":"0f671fd6a820dd96d7a96a9059e80dfbe68ccc7748614b4027b9484bb39dc2ec"} Oct 29 21:59:15.682837 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.682784 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podStartSLOduration=1.6827713389999999 podStartE2EDuration="1.682771339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:15.680041383 +0000 UTC m=+1014.528518254" watchObservedRunningTime="2026-10-29 21:59:15.682771339 +0000 UTC m=+1014.531248195" Oct 29 21:59:15.727767 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:15.727726 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:16.434144 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:59:16.434099 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:59:16.732791 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:16.732768 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:16.989544 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:16.989465 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 21:59:16.994033 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:16.994011 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.019596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.019561 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 21:59:17.025728 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.025697 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 21:59:17.030699 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.030677 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.051260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051225 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.051485 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051455 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.051667 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051507 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfh5\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.051667 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051564 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.051667 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051614 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.051667 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051645 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051817 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051842 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051929 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051956 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.051979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052042 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052024 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.052500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052095 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052181 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052217 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052500 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052267 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hlc\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.052774 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.052714 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 21:59:17.153783 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.153783 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153786 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153801 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153825 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153846 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.153922 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154010 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154044 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154083 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hlc\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154128 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154157 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154193 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfh5\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154300 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154334 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154347 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154346 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.154471 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154412 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.155027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154528 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.155027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154555 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.155027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154629 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.155027 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.154656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.155281 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.155246 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.155341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.155323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.155591 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.155526 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.155591 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.155552 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.155780 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.155706 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.156538 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.156510 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.156640 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.156568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.157011 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.156991 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.157193 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.157175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.182944 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.182896 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.183123 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.183098 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hlc\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.183374 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.183353 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.183416 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.183401 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfh5\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.310205 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.310127 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:17.361450 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.361417 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:17.635270 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.635242 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 21:59:17.636608 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:17.636572 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a6c50c_d11b_4820_b3d2_50af7f22e936.slice/crio-a13e4a3990cb6794808809333b9eb9ae4fbaac961e8ed0815e5c4ee231c07ed3 WatchSource:0}: Error finding container a13e4a3990cb6794808809333b9eb9ae4fbaac961e8ed0815e5c4ee231c07ed3: Status 404 returned error can't find the container with id a13e4a3990cb6794808809333b9eb9ae4fbaac961e8ed0815e5c4ee231c07ed3 Oct 29 21:59:17.639139 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.639099 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:17.639241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.639166 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:17.639241 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.639193 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:17.645259 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.645223 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" event={"ID":"48a6c50c-d11b-4820-b3d2-50af7f22e936","Type":"ContainerStarted","Data":"a13e4a3990cb6794808809333b9eb9ae4fbaac961e8ed0815e5c4ee231c07ed3"} Oct 29 21:59:17.645530 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.645518 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:17.646411 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.646393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 21:59:17.683097 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.683076 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 21:59:17.697177 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:17.697140 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e14c1f_c95e_4af2_b72e_f288fc598938.slice/crio-7afb3b1879afe594e8ed9d3e882e107f6b908a061c0e2bdeda3796d0da6a8046 WatchSource:0}: Error finding container 7afb3b1879afe594e8ed9d3e882e107f6b908a061c0e2bdeda3796d0da6a8046: Status 404 returned error can't find the container with id 7afb3b1879afe594e8ed9d3e882e107f6b908a061c0e2bdeda3796d0da6a8046 Oct 29 21:59:17.703149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.703102 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:17.703583 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.703552 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:17.706318 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:17.706286 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:18.196173 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.196139 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 21:59:18.201948 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.201886 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.223328 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.223299 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 21:59:18.300149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300149 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300167 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln45d\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300361 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300263 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300361 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300434 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300372 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300513 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300563 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300555 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300596 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300585 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.300632 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.300625 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402410 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402378 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402432 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402452 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402476 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402516 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402598 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402544 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln45d\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402613 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402653 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.402877 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402703 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.403107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.402874 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.403107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.403030 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.403107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.403070 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.403260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.403223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.403260 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.403233 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.405037 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.405017 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.405124 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.405043 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.428941 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.428876 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.429463 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.429444 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln45d\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.525576 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.525521 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:18.697325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.691206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" event={"ID":"48a6c50c-d11b-4820-b3d2-50af7f22e936","Type":"ContainerStarted","Data":"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8"} Oct 29 21:59:18.700976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.700858 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" event={"ID":"d4e14c1f-c95e-4af2-b72e-f288fc598938","Type":"ContainerStarted","Data":"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b"} Oct 29 21:59:18.700976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.700926 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" event={"ID":"d4e14c1f-c95e-4af2-b72e-f288fc598938","Type":"ContainerStarted","Data":"7afb3b1879afe594e8ed9d3e882e107f6b908a061c0e2bdeda3796d0da6a8046"} Oct 29 21:59:18.782985 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.760059 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podStartSLOduration=2.760042753 podStartE2EDuration="2.760042753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:18.758730529 +0000 UTC m=+1017.607207387" watchObservedRunningTime="2026-10-29 21:59:18.760042753 +0000 UTC m=+1017.608519599" Oct 29 21:59:18.915463 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.915416 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podStartSLOduration=2.915400915 podStartE2EDuration="2.915400915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:18.820173621 +0000 UTC m=+1017.668650469" watchObservedRunningTime="2026-10-29 21:59:18.915400915 +0000 UTC m=+1017.763877760" Oct 29 21:59:18.917236 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.917214 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 21:59:18.917981 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:18.917953 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd006d9b8_4c3d_474d_8667_9779d742c8ff.slice/crio-e539f858e916d705b2f68f052151c8859f75b3c60b56aa03fe03594455282977 WatchSource:0}: Error finding container e539f858e916d705b2f68f052151c8859f75b3c60b56aa03fe03594455282977: Status 404 returned error can't find the container with id e539f858e916d705b2f68f052151c8859f75b3c60b56aa03fe03594455282977 Oct 29 21:59:18.920492 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.920462 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:18.920564 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.920522 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:18.920564 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:18.920551 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:19.310788 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.310757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:19.311809 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.311781 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:19.311947 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.311839 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:19.364308 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.364230 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:19.368309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.367925 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:19.368309 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.368038 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:19.708957 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.708927 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" event={"ID":"d006d9b8-4c3d-474d-8667-9779d742c8ff","Type":"ContainerStarted","Data":"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f"} Oct 29 21:59:19.709129 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:19.708972 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" event={"ID":"d006d9b8-4c3d-474d-8667-9779d742c8ff","Type":"ContainerStarted","Data":"e539f858e916d705b2f68f052151c8859f75b3c60b56aa03fe03594455282977"} Oct 29 21:59:20.326423 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.319021 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.326423 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.319086 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:20.375928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.359476 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podStartSLOduration=2.359455633 podStartE2EDuration="2.359455633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:19.777018981 +0000 UTC m=+1018.625495843" watchObservedRunningTime="2026-10-29 21:59:20.359455633 +0000 UTC m=+1019.207932480" Oct 29 21:59:20.375928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.373644 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.375928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.373700 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:20.525936 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.525180 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 21:59:20.670975 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.664038 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.670975 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.664191 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:20.685252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.685222 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:20.726930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.709027 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:20.791930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.773363 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8867e7d-dockercfg-c96bh\"" Oct 29 21:59:20.791930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.785603 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 21:59:20.837411 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.794212 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8867e7d\"" Oct 29 21:59:20.999548 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:20.960811 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.017341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.017292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctcxn\" (UniqueName: \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.124011 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.123976 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctcxn\" (UniqueName: \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.124352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.124322 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.125353 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.125322 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.196981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.195131 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctcxn\" (UniqueName: \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.316788 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.316752 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.317066 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.317035 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:21.381927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.378051 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.381927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.378110 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:21.426830 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.426803 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8867e7d-dockercfg-c96bh\"" Oct 29 21:59:21.431483 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.431465 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:21.527089 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.527061 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.527171 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:21.527120 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:22.214227 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.213339 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 21:59:22.217980 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:22.217955 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c7dc9b_11b6_47a1_b8d8_c76a71b0c3af.slice/crio-24445e108b825c7e277351b6e958579febcbdcaefaca3d92f12ec3ea21051b8e WatchSource:0}: Error finding container 24445e108b825c7e277351b6e958579febcbdcaefaca3d92f12ec3ea21051b8e: Status 404 returned error can't find the container with id 24445e108b825c7e277351b6e958579febcbdcaefaca3d92f12ec3ea21051b8e Oct 29 21:59:22.312860 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.312347 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.312860 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.312478 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:22.367931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.367629 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.367931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.367703 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:22.528698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.528582 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.528698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.528640 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:22.968986 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.960859 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" event={"ID":"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af","Type":"ContainerStarted","Data":"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640"} Oct 29 21:59:22.972924 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.971114 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:22.972924 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:22.971160 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" event={"ID":"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af","Type":"ContainerStarted","Data":"24445e108b825c7e277351b6e958579febcbdcaefaca3d92f12ec3ea21051b8e"} Oct 29 21:59:23.018255 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.018209 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" podStartSLOduration=3.018197449 podStartE2EDuration="3.018197449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:23.014977678 +0000 UTC m=+1021.863454525" watchObservedRunningTime="2026-10-29 21:59:23.018197449 +0000 UTC m=+1021.866674294" Oct 29 21:59:23.319984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.317281 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.319984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.317347 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:23.371130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.370181 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.371130 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.370236 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:23.527088 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.527048 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.527240 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:23.527110 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:24.322933 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.320102 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.322933 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.320160 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:24.362304 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.362267 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.362592 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.362562 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:24.533196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.533162 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.533487 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:24.533444 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:25.321923 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.316854 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.321923 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.316931 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:25.395928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.382041 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.395928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.382106 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 21:59:25.541922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.538012 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.541922 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:25.538075 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:26.370930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:26.359272 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.389030 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:26.388987 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 21:59:26.651571 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:26.651465 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.666435 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:26.652187 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:27.021939 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.016646 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 21:59:27.384984 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.375011 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.409814 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.409773 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:27.410118 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.410085 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.453234 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.452002 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.651981 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.647581 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.667107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.667003 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:27.667107 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.667087 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:27.667352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.667169 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.667352 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.667223 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 21:59:27.714573 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.714527 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.757482 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:27.757438 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 21:59:28.072928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.042774 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.072928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.043215 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:59:28.378982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.378017 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.378982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.378815 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:59:28.409925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.379811 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:28.409925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.379857 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.454925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.452320 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.454925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.452753 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:59:28.469931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.457157 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:28.469931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.457219 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.664023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.621008 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:28.664023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.621078 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:28.664023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.648402 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 21:59:28.664023 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.648837 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:59:28.822019 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.821695 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.841930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.838939 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:28.841930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.840487 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:28.890698 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:28.870248 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 21:59:29.033451 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.033353 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 21:59:29.053786 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.053746 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.142206 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.142172 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 21:59:29.387683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.387582 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.532989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.529202 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:29.532989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.529288 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.532989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.529883 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:59:29.532989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.529951 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 21:59:29.546129 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:29.546095 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:30.101934 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.101517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 21:59:30.283556 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.283524 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 21:59:30.458395 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.458366 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:30.459621 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.459600 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:30.460456 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.460440 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 21:59:30.703700 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.703666 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:30.704526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.704498 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:30.705426 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:30.705400 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 21:59:33.066542 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.066503 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 21:59:33.080640 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.080614 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.112597 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.112573 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4082f-openshift-default-dockercfg-fp7dc\"" Oct 29 21:59:33.151283 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.151252 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 21:59:33.159018 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.158987 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159144 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159071 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159200 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159268 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159235 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2lw\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159287 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159327 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159317 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159419 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159387 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159535 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159515 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.159602 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.159542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.269050 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.266411 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.269050 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.266518 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.271662 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.271570 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.275926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.275057 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2lw\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.275926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.275124 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.275926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.275153 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.275926 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.275344 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.294976 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.294365 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.296242 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.296212 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.300039 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.299994 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.300238 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.300218 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.300647 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.300574 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.301998 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.301976 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.302378 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.302275 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.386109 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.383804 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2lw\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.394975 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.390701 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.394975 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.392927 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.403752 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.398779 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\") pod \"e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.604567 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.604528 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 21:59:33.613137 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.613107 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 21:59:33.622268 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.622157 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630032 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.629997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630046 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630132 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630218 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630186 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9jp\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630224 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630405 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630370 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630556 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630473 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.630610 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.630578 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.694565 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.694534 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:33.732191 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732157 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732212 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732245 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732292 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732341 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732327 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9jp\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732351 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732422 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732515 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732475 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732563 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732682 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732583 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732827 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732801 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.732883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.732846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.733083 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.733058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.733083 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.733073 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.735098 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.735069 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.735758 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.735738 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.766000 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.765968 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9jp\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.768206 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.768179 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:33.952049 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:33.952027 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:34.014282 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.014255 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 21:59:34.020495 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.020266 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 21:59:34.043931 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.043866 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.044113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.044051 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.044113 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.044108 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.262274 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:34.262243 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc7edf8_edf6_45fc_aea4_4228f96c231b.slice/crio-d8785afc25939685d4f8cbec6eb38bad26a5c1bac83fb74256076b619cb1f2bb WatchSource:0}: Error finding container d8785afc25939685d4f8cbec6eb38bad26a5c1bac83fb74256076b619cb1f2bb: Status 404 returned error can't find the container with id d8785afc25939685d4f8cbec6eb38bad26a5c1bac83fb74256076b619cb1f2bb Oct 29 21:59:34.266187 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.266148 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.266299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.266234 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.266299 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.266279 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:34.267094 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.267061 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 21:59:34.545990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.545860 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" event={"ID":"5bc7edf8-edf6-45fc-aea4-4228f96c231b","Type":"ContainerStarted","Data":"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f"} Oct 29 21:59:34.545990 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.545934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" event={"ID":"5bc7edf8-edf6-45fc-aea4-4228f96c231b","Type":"ContainerStarted","Data":"d8785afc25939685d4f8cbec6eb38bad26a5c1bac83fb74256076b619cb1f2bb"} Oct 29 21:59:34.548254 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.548218 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerStarted","Data":"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24"} Oct 29 21:59:34.548404 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.548275 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerStarted","Data":"3ceb4cdde67fc6b9bb4b18a3d7e2f419598e25b3aa67e27b5ff584ab7625b407"} Oct 29 21:59:34.597526 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.597475 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podStartSLOduration=1.597458317 podStartE2EDuration="1.597458317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:34.595361636 +0000 UTC m=+1033.443838481" watchObservedRunningTime="2026-10-29 21:59:34.597458317 +0000 UTC m=+1033.445935164" Oct 29 21:59:34.651718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.651668 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podStartSLOduration=1.651654696 podStartE2EDuration="1.651654696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:34.648855669 +0000 UTC m=+1033.497332515" watchObservedRunningTime="2026-10-29 21:59:34.651654696 +0000 UTC m=+1033.500131545" Oct 29 21:59:34.695048 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.695021 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:34.952503 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:34.952458 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:35.700586 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:35.700553 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:35.959683 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:35.959653 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:35.959853 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:35.959708 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:36.575673 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:36.575638 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:36.576439 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:36.576413 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 21:59:36.959923 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:36.957467 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:36.959923 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:36.957526 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:37.953653 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:37.953614 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:37.954325 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:37.953792 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:38.633930 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.627853 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 21:59:38.646928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.642124 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.671716 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.668264 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8867e7d-serving-cert\"" Oct 29 21:59:38.715518 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.714554 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 21:59:38.766929 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.748500 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn2t\" (UniqueName: \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.767214 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.767186 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.767552 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.767529 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.768377 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.768143 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.898977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.898092 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.898977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.898207 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn2t\" (UniqueName: \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.898977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.898244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.898977 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.898396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.922927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.922776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.926923 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.925769 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.939988 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.939497 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:38.990927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.987445 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:38.990927 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:38.987584 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:39.012612 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:39.000807 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn2t\" (UniqueName: \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:39.075251 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:39.075216 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:39.387759 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:39.387728 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 21:59:39.389829 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:39.389808 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:59:39.694133 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:39.688840 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" event={"ID":"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f","Type":"ContainerStarted","Data":"897132697fb6d90033023d17a3c132901aad767d7d4b727d0a0d3345c1756abf"} Oct 29 21:59:40.046956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.044445 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:40.046956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.044508 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:40.696849 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.696676 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" event={"ID":"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f","Type":"ContainerStarted","Data":"59d0cb1c8ed031c875759f98fc6f486824a32ca7e7aaacbb18d77f17c060a789"} Oct 29 21:59:40.696849 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.696831 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:40.744993 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.744936 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" podStartSLOduration=2.394348075 podStartE2EDuration="2.74492115s" totalImagesPullingTime="350.573075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:40.742845125 +0000 UTC m=+1039.591321995" watchObservedRunningTime="2026-10-29 21:59:40.74492115 +0000 UTC m=+1039.593397990" Oct 29 21:59:40.974928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.956405 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:40.974928 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:40.956466 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:41.953114 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:41.953075 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:59:41.953599 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:41.953140 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 21:59:42.992399 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:42.992365 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:42.995364 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:42.995342 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:43.019232 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:43.019201 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 21:59:46.522173 ip-10-0-141-100 kubenswrapper[4376]: E1029 21:59:46.522127 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 21:59:46.722892 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:46.722860 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 21:59:49.974656 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:49.967773 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 21:59:49.982028 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:49.981991 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:49.992234 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:49.992203 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 21:59:50.062508 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.062462 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.062697 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.062612 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.063137 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.062682 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068440 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068403 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068465 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcg8\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068504 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068540 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068611 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068577 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.068815 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.068685 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170392 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170341 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170392 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170398 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcg8\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170533 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170574 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170635 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170597 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170813 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170666 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.170875 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.170834 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171021 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171001 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171022 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171165 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171108 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171289 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171250 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171344 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171299 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171381 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171349 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.171504 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.171481 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.173064 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.173046 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.173285 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.173267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.197177 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.197132 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcg8\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.197388 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.197369 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.311980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.311856 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:50.639120 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.639084 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 21:59:50.640179 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:50.640150 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f10461_40a5_4563_8bcc_e2ab4780f76a.slice/crio-ce6016effce2e996f71caa43fe096224b3cebd108ff1a0a4025f65ce3ac833d9 WatchSource:0}: Error finding container ce6016effce2e996f71caa43fe096224b3cebd108ff1a0a4025f65ce3ac833d9: Status 404 returned error can't find the container with id ce6016effce2e996f71caa43fe096224b3cebd108ff1a0a4025f65ce3ac833d9 Oct 29 21:59:50.643054 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.643017 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:50.643132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.643095 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:50.643132 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.643127 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:50.827792 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.827746 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" event={"ID":"a1f10461-40a5-4563-8bcc-e2ab4780f76a","Type":"ContainerStarted","Data":"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727"} Oct 29 21:59:50.827989 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.827810 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" event={"ID":"a1f10461-40a5-4563-8bcc-e2ab4780f76a","Type":"ContainerStarted","Data":"ce6016effce2e996f71caa43fe096224b3cebd108ff1a0a4025f65ce3ac833d9"} Oct 29 21:59:50.878602 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:50.878547 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podStartSLOduration=1.878530836 podStartE2EDuration="1.878530836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:50.876538959 +0000 UTC m=+1049.725015806" watchObservedRunningTime="2026-10-29 21:59:50.878530836 +0000 UTC m=+1049.727007682" Oct 29 21:59:51.312104 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:51.312067 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:51.314021 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:51.313994 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:51.314162 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:51.314045 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:52.313072 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:52.313035 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:52.313592 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:52.313097 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:53.312675 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:53.312635 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:53.312870 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:53.312696 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:54.052477 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.052420 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 21:59:54.091213 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.091171 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 21:59:54.091663 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.091641 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.101987 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.101953 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.105982 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.105947 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.106498 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.106466 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.106652 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.106630 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.106749 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.106730 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.106850 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.106832 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5k2l\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.107007 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.106987 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.107252 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.107229 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.107376 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.107363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.208744 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.208697 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.208983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.208851 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.208983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.208934 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.208983 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.208969 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209042 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209097 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209158 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209138 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209155 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k2l\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209185 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209250 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209314 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209303 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209522 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209366 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209578 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209559 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.209980 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.209960 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.211468 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.211448 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.211520 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.211452 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.261169 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.261133 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5k2l\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.261353 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.261171 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.312997 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.312891 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:54.312997 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.312964 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:54.421745 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.421704 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:54.773569 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.773539 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 21:59:54.774564 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:54.774539 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8676ade_b183_4ada_9a0a_94122c52b341.slice/crio-c73105f931cf5a0e20332efabff5a706f753fa0e4c50225b6cc0221cfecd4dd5 WatchSource:0}: Error finding container c73105f931cf5a0e20332efabff5a706f753fa0e4c50225b6cc0221cfecd4dd5: Status 404 returned error can't find the container with id c73105f931cf5a0e20332efabff5a706f753fa0e4c50225b6cc0221cfecd4dd5 Oct 29 21:59:54.777404 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.777372 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:54.777548 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.777444 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:54.777548 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.777472 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 21:59:54.872196 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:54.872159 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" event={"ID":"a8676ade-b183-4ada-9a0a-94122c52b341","Type":"ContainerStarted","Data":"c73105f931cf5a0e20332efabff5a706f753fa0e4c50225b6cc0221cfecd4dd5"} Oct 29 21:59:55.175275 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.175185 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 21:59:55.180675 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.180647 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.207718 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.204508 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 21:59:55.212279 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.212243 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d4082f-serving-cert\"" Oct 29 21:59:55.224710 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.224438 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.224710 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.224539 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlh99\" (UniqueName: \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.224710 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.224575 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.225584 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.225234 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.312891 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.312857 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:55.313081 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.312951 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:55.326539 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.326503 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.326752 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.326717 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.326823 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.326791 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlh99\" (UniqueName: \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.326879 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.326823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.327588 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.327563 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.329760 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.329731 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.329890 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.329810 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.367966 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.367934 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlh99\" (UniqueName: \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\") pod \"maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.509518 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.495941 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:55.893769 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.886855 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" event={"ID":"a8676ade-b183-4ada-9a0a-94122c52b341","Type":"ContainerStarted","Data":"f9330b7e0a3f2e6ef4b34e4fa2edb9907ed73268205438505a6bbe74f3a55d4e"} Oct 29 21:59:55.993655 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.993602 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podStartSLOduration=1.993586327 podStartE2EDuration="1.993586327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.947375318 +0000 UTC m=+1054.795852165" watchObservedRunningTime="2026-10-29 21:59:55.993586327 +0000 UTC m=+1054.842063172" Oct 29 21:59:55.996865 ip-10-0-141-100 kubenswrapper[4376]: W1029 21:59:55.996834 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8467d658_5ac1_49de_ba22_d4dde48f1c57.slice/crio-b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5 WatchSource:0}: Error finding container b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5: Status 404 returned error can't find the container with id b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5 Oct 29 21:59:55.998105 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:55.998079 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 21:59:56.316925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:56.313216 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.316925 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:56.313302 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:56.457408 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:56.450835 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 21:59:56.483106 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:56.483049 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.483400 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:56.483362 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 21:59:57.124504 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.124463 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" event={"ID":"8467d658-5ac1-49de-ba22-d4dde48f1c57","Type":"ContainerStarted","Data":"10b301f17bdc01ee673bc45e74d74c452e2133898d219737a16704f1383debeb"} Oct 29 21:59:57.124773 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.124753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" event={"ID":"8467d658-5ac1-49de-ba22-d4dde48f1c57","Type":"ContainerStarted","Data":"b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5"} Oct 29 21:59:57.124883 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.124871 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 21:59:57.191715 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.191649 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" podStartSLOduration=1.705756913 podStartE2EDuration="2.19162761s" totalImagesPullingTime="485.870697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:57.188488726 +0000 UTC m=+1056.036965594" watchObservedRunningTime="2026-10-29 21:59:57.19162761 +0000 UTC m=+1056.040104457" Oct 29 21:59:57.312735 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.312694 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.312956 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.312755 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:57.436462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.426638 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.436462 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:57.426711 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 21:59:58.312509 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:58.312470 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.312707 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:58.312544 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 21:59:58.422448 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:58.422415 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.422691 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:58.422669 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 21:59:59.318001 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:59.317967 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:59.318421 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:59.318269 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:59.319110 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:59.319091 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 21:59:59.422436 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:59.422393 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:59.422630 ip-10-0-141-100 kubenswrapper[4376]: I1029 21:59:59.422465 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 22:00:00.422555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:00.422492 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:00:00.423248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:00.423212 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 22:00:01.422634 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.422586 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:00:01.423080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.422653 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 22:00:01.705683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.705643 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:00:01.710375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.710345 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.737152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.737113 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:00:01.786549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786510 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.786751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786718 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.786810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786778 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.786870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786853 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.787041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786921 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zln9s\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.787041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.786972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.787423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.787397 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.787544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.787514 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.787624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.787608 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.889813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.889762 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.889842 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.889898 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.889963 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zln9s\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890012 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890073 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890143 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890218 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.890437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890393 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.891013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.890986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.891138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.891049 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.891138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.891064 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.891346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.891320 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.892423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.892396 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.894298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.894275 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.916667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.916633 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:01.916810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:01.916711 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zln9s\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:02.043185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.043093 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:02.446003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.445917 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:00:02.446564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.446526 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 29 22:00:02.588434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.588402 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:00:02.589285 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:00:02.589249 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec82d2_aa23_46ac_a103_fa51555c216b.slice/crio-369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a WatchSource:0}: Error finding container 369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a: Status 404 returned error can't find the container with id 369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a Oct 29 22:00:02.592320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.592283 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:02.592416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.592358 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:02.592416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:02.592390 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:03.178696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:03.178609 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 22:00:03.343977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:03.343500 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" event={"ID":"90ec82d2-aa23-46ac-a103-fa51555c216b","Type":"ContainerStarted","Data":"a422545cf321c68715f0b2519b37323a0b45396e4e27dc21742b16aa57be2a28"} Oct 29 22:00:03.343977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:03.343578 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" event={"ID":"90ec82d2-aa23-46ac-a103-fa51555c216b","Type":"ContainerStarted","Data":"369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a"} Oct 29 22:00:03.399717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:03.398285 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podStartSLOduration=2.398268706 podStartE2EDuration="2.398268706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.395481355 +0000 UTC m=+1062.243958303" watchObservedRunningTime="2026-10-29 22:00:03.398268706 +0000 UTC m=+1062.246745551" Oct 29 22:00:04.032653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.031305 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 22:00:04.043934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.043438 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 22:00:04.044138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.043955 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:04.052945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.049173 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 22:00:04.052945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.049284 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:04.052945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.049327 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:04.588926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:04.579411 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.044841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.044784 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:05.054055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.054001 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:05.153072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.153021 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.428467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.426431 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 29 22:00:05.438661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.438614 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:00:05.551923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.551333 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.593090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.593036 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.670111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.670065 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.710951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.708995 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:05.756270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:05.756207 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.063098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.046078 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.063098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.047390 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:06.063098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.051318 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:06.063098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.051388 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:06.202382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.202324 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.242896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.242856 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:00:06.328989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.315994 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.441113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.441040 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.447694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.447651 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:06.560415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.559009 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.560415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.559710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:06.663995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.629018 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.678930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.665254 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:00:06.967006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:06.966976 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:07.086845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.086798 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:00:07.091602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.091568 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:00:07.188062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.180874 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:07.188062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.180954 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:07.394012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.391433 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.528916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.527279 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:00:07.589928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.585145 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.665875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.665759 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.843091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.843032 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.970050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:07.969999 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.089086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.089031 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:00:08.103728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.103678 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.121494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.115731 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:08.121494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.115811 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:08.431010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.428360 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:08.532981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.531091 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.675249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.675166 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.828708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.828659 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 29 22:00:08.971461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.927018 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:00:08.971461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.927644 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:00:09.000749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.997744 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:09.000749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:08.998482 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:09.062388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.062343 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:09.062693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.062647 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:09.101253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.098195 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:09.101253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.099773 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:09.223928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.117521 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:09.570980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.565997 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:09.609735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.609679 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:09.704928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.704283 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:00:09.733936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.733110 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:09.733936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.733196 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:00:09.738005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.735837 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:09.784170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.767058 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:00:09.784170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.767139 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:09.814001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.813882 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:00:09.815107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:09.815081 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:00:10.100696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.100648 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:10.101020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.100972 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:10.588882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.588832 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:00:10.595098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.595068 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.602069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.602037 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:00:10.612754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.612724 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d7f9e0-serving-cert\"" Oct 29 22:00:10.616858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.616827 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.617075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.616875 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.617075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.616954 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.617075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.616985 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzp6\" (UniqueName: \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.718999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.718943 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzp6\" (UniqueName: \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.719311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.719293 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.719409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.719347 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.719479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.719440 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.719479 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:00:10.719463 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-serving-cert: secret "maas-api-e2e-api-b-d7f9e0-serving-cert" not found Oct 29 22:00:10.719578 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:00:10.719527 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls podName:82fe5a85-b397-4875-8003-9c14f662c02b nodeName:}" failed. No retries permitted until 2026-10-29 22:00:11.219512137 +0000 UTC m=+1070.067988962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls") pod "maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" (UID: "82fe5a85-b397-4875-8003-9c14f662c02b") : secret "maas-api-e2e-api-b-d7f9e0-serving-cert" not found Oct 29 22:00:10.720350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.720316 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.722161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.722134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:10.832792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:10.832746 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzp6\" (UniqueName: \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:11.068413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:11.065054 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:11.068413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:11.065118 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:11.256932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:11.255558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:11.277135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:11.277094 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") pod \"maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:11.515277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:11.515225 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:12.047182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:12.046320 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:12.047182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:12.046392 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:12.179595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:12.171190 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:00:13.044515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.044471 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:00:13.044687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.044544 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:00:13.220183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.220136 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" event={"ID":"82fe5a85-b397-4875-8003-9c14f662c02b","Type":"ContainerStarted","Data":"c11593273ec5a29f5c74572b96f02c0610f6a22d97626de582cca1ba867e8960"} Oct 29 22:00:13.220724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.220204 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" event={"ID":"82fe5a85-b397-4875-8003-9c14f662c02b","Type":"ContainerStarted","Data":"b8191ccbf3ef67af98aab6e872fc19caa09cfdcf343ad17b16c65e2ccf24a3b5"} Oct 29 22:00:13.220724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.220226 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:13.269023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:13.268950 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" podStartSLOduration=2.846633513 podStartE2EDuration="3.268934456s" totalImagesPullingTime="422.300943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:13.26563418 +0000 UTC m=+1072.114111038" watchObservedRunningTime="2026-10-29 22:00:13.268934456 +0000 UTC m=+1072.117411301" Oct 29 22:00:14.137211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:14.137179 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:14.137649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:14.137588 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:14.139686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:14.139660 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:00:16.441812 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:00:16.441772 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:00:16.544228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:16.531750 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:16.544228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:16.531823 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:18.258795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.258759 4376 generic.go:356] "Generic (PLEG): container finished" podID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerID="5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba" exitCode=0 Oct 29 22:00:18.259314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.258837 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerDied","Data":"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba"} Oct 29 22:00:18.259373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.259359 4376 scope.go:122] "RemoveContainer" containerID="5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba" Oct 29 22:00:18.261165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.261134 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:18.261263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.261209 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:18.261263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:18.261241 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:19.258238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:19.258193 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:00:19.273602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:19.273566 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerStarted","Data":"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed"} Oct 29 22:00:19.654435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:19.654333 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:00:19.654767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:19.654744 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:19.654943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:19.654891 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:20.659483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:20.659441 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:20.660032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:20.659507 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:21.656127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:21.656088 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:21.656409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:21.656381 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:22.651376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:22.651325 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:22.651958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:22.651401 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:23.674927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:23.674694 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:23.674927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:23.674770 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:24.648540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:24.648502 4376 patch_prober.go:29] interesting pod/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:00:24.648729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:24.648561 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:00:25.669930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:25.668210 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:00:25.670454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:25.670102 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:00:25.678864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:25.678826 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:00:29.257257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.257222 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:00:29.263745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.263718 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.298202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.298168 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-579f09-openshift-default-dockercfg-jwwl9\"" Oct 29 22:00:29.306332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.306293 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:00:29.319390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.319347 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.319603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.319410 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvm7\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.319685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.319616 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.319685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.319661 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.319807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.319694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.320118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.320094 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.320393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.320370 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.320514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.320441 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.320581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.320547 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.437036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.436997 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.445177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.445143 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.459930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.445856 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.460333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.460282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.460895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.460872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.461099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.461079 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvm7\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.461297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.461277 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.461446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.461428 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.461574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.461558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.462165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.462144 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.462301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.441567 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.462722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.457691 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.467937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.460781 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.468114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.465263 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.468251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.465866 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.478320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.475543 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.527989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.526569 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.527989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.527723 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvm7\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\") pod \"e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:29.594041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:29.593220 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:30.067784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.067752 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:00:30.068752 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:00:30.068726 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf40a427_273f_4d62_96fc_e866a89e9726.slice/crio-2c5dba2dcf9c4246517c5fb242562867a188c01a525f812628b2d34d1bdd80dd WatchSource:0}: Error finding container 2c5dba2dcf9c4246517c5fb242562867a188c01a525f812628b2d34d1bdd80dd: Status 404 returned error can't find the container with id 2c5dba2dcf9c4246517c5fb242562867a188c01a525f812628b2d34d1bdd80dd Oct 29 22:00:30.071499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.071453 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:30.071611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.071543 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:30.071611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.071591 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:30.454818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.454781 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerStarted","Data":"da5113c9ecbf316d808f3a989b1427271ebf95f6de3e2c43019727f71a97d280"} Oct 29 22:00:30.455353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.454841 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerStarted","Data":"2c5dba2dcf9c4246517c5fb242562867a188c01a525f812628b2d34d1bdd80dd"} Oct 29 22:00:30.507608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.507544 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podStartSLOduration=1.507529761 podStartE2EDuration="1.507529761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:30.504014631 +0000 UTC m=+1089.352491479" watchObservedRunningTime="2026-10-29 22:00:30.507529761 +0000 UTC m=+1089.356006608" Oct 29 22:00:30.593689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:30.593653 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:31.599801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:31.599773 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:32.469398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:32.469365 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:32.470349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:32.470330 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:00:44.210058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.209999 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.236033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.235980 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.412988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.410172 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.536141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.536020 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.712120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.712057 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:00:44.727060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.727007 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.756231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.756181 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.828356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:44.828212 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.257298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.257245 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.259258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.259231 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:00:45.470929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.457866 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.561463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.561339 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.592667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.592621 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.739941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.739703 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:00:45.755995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.744436 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:00:45.755995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.744537 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:45.755995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.744602 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:00:45.800215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.800169 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:00:45.843096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.831315 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.872380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:45.871865 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.133019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.132058 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:46.374808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.366623 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:00:46.374808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.367038 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 29 22:00:46.428983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.402247 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:00:46.428983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.404421 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:46.467532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.466255 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.535150 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.535110 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:46.538154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.538113 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.632079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.626352 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.838937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.824267 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.877135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.877065 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.990774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.990136 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.077929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:46.855257 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:00:47.250600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.225323 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j5t99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:47.250600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.249650 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.433254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.421627 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.514330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.514049 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:47.836102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.835994 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.896859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.839987 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:00:47.931581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:47.863322 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:48.019168 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:00:47.961547 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:00:48.050941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.037217 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:00:48.057266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.057218 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.286381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.277701 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.427430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.427382 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:00:48.506959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.504063 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.574223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.534739 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:48.812945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.795775 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.964008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:48.905283 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:00:49.092930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.092099 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:49.092930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.092512 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:00:49.292996 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.291827 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:49.508932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.508851 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:00:49.518815 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.514646 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:49.539252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.539193 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:49.555170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.554841 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:00:49.555170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.554937 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:49.955025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:49.943599 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:00:50.030055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.030024 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:00:50.032412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.032384 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:00:50.097337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.096585 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:50.102244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.102210 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:00:50.124934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.122540 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:00:50.130565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:50.130537 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:00:55.940023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:55.939983 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:00:55.945766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:55.945745 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:55.965978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:55.965938 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:00:55.974394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:55.974356 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-579f09-serving-cert\"" Oct 29 22:00:56.006575 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.006535 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.006785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.006645 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.006869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.006845 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb9z\" (UniqueName: \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.007406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.007232 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.108527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.108479 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.108742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.108583 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.108830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.108753 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb9z\" (UniqueName: \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.108894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.108859 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.109311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.109283 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.111538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.111514 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.111933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.111893 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.135768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.135718 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb9z\" (UniqueName: \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\") pod \"maas-api-e2e-api-b-579f09-755f848844-d8lk7\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.264403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.264358 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:56.665158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:56.665119 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:00:56.666940 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:00:56.666895 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb13f771_8a74_4195_871a_9a0a4fb1b454.slice/crio-5e0fe25c7fcf5ee5e56a6215c1cd1ae4a39147a94ddb747542ea846e0702a7e1 WatchSource:0}: Error finding container 5e0fe25c7fcf5ee5e56a6215c1cd1ae4a39147a94ddb747542ea846e0702a7e1: Status 404 returned error can't find the container with id 5e0fe25c7fcf5ee5e56a6215c1cd1ae4a39147a94ddb747542ea846e0702a7e1 Oct 29 22:00:57.489126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.489086 4376 generic.go:356] "Generic (PLEG): container finished" podID="2dd67725-79b9-499b-a238-61d27d1db363" containerID="3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23" exitCode=0 Oct 29 22:00:57.489662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.489166 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerDied","Data":"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23"} Oct 29 22:00:57.489732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.489707 4376 scope.go:122] "RemoveContainer" containerID="3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23" Oct 29 22:00:57.491274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491251 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" event={"ID":"cb13f771-8a74-4195-871a-9a0a4fb1b454","Type":"ContainerStarted","Data":"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d"} Oct 29 22:00:57.491422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491289 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" event={"ID":"cb13f771-8a74-4195-871a-9a0a4fb1b454","Type":"ContainerStarted","Data":"5e0fe25c7fcf5ee5e56a6215c1cd1ae4a39147a94ddb747542ea846e0702a7e1"} Oct 29 22:00:57.491422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491341 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:00:57.491525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491448 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:57.491582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491530 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:57.491634 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.491584 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:00:57.610861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:57.610790 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" podStartSLOduration=2.033106334 podStartE2EDuration="2.61077072s" totalImagesPullingTime="577.664386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:57.598624636 +0000 UTC m=+1116.447101478" watchObservedRunningTime="2026-10-29 22:00:57.61077072 +0000 UTC m=+1116.459247569" Oct 29 22:00:58.499696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:58.499655 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerStarted","Data":"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e"} Oct 29 22:00:58.727090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:58.727047 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:00:58.728195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:58.728166 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:00:58.728312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:58.728219 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:00:59.734926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:59.729622 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:00:59.734926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:00:59.729687 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:01:00.727262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:00.727225 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:01:00.727445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:00.727276 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:01:01.727166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:01.727129 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:01:01.727565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:01.727190 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:01:02.651940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:02.651022 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:01:02.727047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:02.727012 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:01:02.727200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:02.727069 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:01:03.505472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:03.505446 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:01:03.727527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:03.727482 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:01:03.727965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:03.727566 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:01:04.732612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:04.732580 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:01:04.733509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:04.733486 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:01:16.434840 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:01:16.434802 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:01:22.964927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:22.964012 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.115709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:23.115662 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.296092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:23.295990 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.455843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:23.455799 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.482072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:23.482029 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:33.569788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.569747 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:01:33.575581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.575556 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.597033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.596995 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:01:33.605542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.605513 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5fd295-openshift-default-dockercfg-8sdkg\"" Oct 29 22:01:33.658654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.658621 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.658871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.658769 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.658871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.658857 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.658957 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.658995 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqb9p\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.659215 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.659363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.659413 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.659545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.659526 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763279 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763439 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763510 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763569 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763675 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.763925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.763711 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.772929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.771393 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.772929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.771593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.772929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.771632 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqb9p\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.772929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.772382 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.804928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.773336 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.804928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.785637 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.804928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.786340 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.804928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.798764 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.804928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.804194 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.828979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.826452 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.842988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.838624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqb9p\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\") pod \"e2e-ipp-a-5fd295-openshift-default-7847778489-77c25\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:33.902972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:33.895276 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:34.852795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:34.852747 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:34.853294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:34.852839 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:34.853294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:34.852889 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:34.867668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:34.867635 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:01:35.025580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:35.025534 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerStarted","Data":"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872"} Oct 29 22:01:35.025736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:35.025632 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerStarted","Data":"0a03238dfa847da579e78e9f3197811e93e7c43418d24b9e6306735d5b670acd"} Oct 29 22:01:35.076256 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:35.076155 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podStartSLOduration=2.076140983 podStartE2EDuration="2.076140983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:35.074727942 +0000 UTC m=+1153.923204789" watchObservedRunningTime="2026-10-29 22:01:35.076140983 +0000 UTC m=+1153.924617828" Oct 29 22:01:35.911928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:35.910044 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:35.938315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:35.938283 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:36.044318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:36.044294 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:36.046391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:36.046302 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:01:46.604927 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:01:46.603475 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:01:49.050062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.050026 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 22:01:49.050649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.050272 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" containerID="cri-o://1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed" gracePeriod=30 Oct 29 22:01:49.790920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.790874 4376 generic.go:356] "Generic (PLEG): container finished" podID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerID="e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24" exitCode=0 Oct 29 22:01:49.791134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.790939 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerDied","Data":"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24"} Oct 29 22:01:49.791490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.791471 4376 scope.go:122] "RemoveContainer" containerID="e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24" Oct 29 22:01:49.793205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.793175 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:49.793294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.793245 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:49.793294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:49.793280 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:01:50.801110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:50.801074 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerStarted","Data":"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47"} Oct 29 22:01:51.575861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:51.575820 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:01:51.695463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:51.695422 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:01:51.696677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:51.696648 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:01:51.696823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:51.696723 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:01:52.695026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:52.694985 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:01:52.695533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:52.695046 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:01:53.695915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:53.695866 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:01:53.696336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:53.695939 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:01:54.387253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.387226 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:01:54.609685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.609581 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.616842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.616810 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617123 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617505 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617669 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617813 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.617954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.617941 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.618062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.618050 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\") pod \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\" (UID: \"d81945ba-078c-49ff-a4ef-60ec6ed2e21e\") " Oct 29 22:01:54.618399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.613112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "kube-api-access-9qnrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:54.618571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.618544 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:01:54.618789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.618766 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:01:54.618963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.618937 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:01:54.619120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.619099 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:54.619371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.619282 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:01:54.619984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.619967 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnrq\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-kube-api-access-9qnrq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.620114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.620104 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.620179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.620170 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.620259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.620246 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.620333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.620313 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.620396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.620337 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.621442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.621416 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:01:54.621553 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.621475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:54.621834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.621811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy" pod "d81945ba-078c-49ff-a4ef-60ec6ed2e21e" (UID: "d81945ba-078c-49ff-a4ef-60ec6ed2e21e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:01:54.695673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.695642 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:01:54.695811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.695700 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:01:54.722037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.722013 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.722037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.722036 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.722419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.722045 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81945ba-078c-49ff-a4ef-60ec6ed2e21e-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:54.871639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.871546 4376 generic.go:356] "Generic (PLEG): container finished" podID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerID="1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed" exitCode=0 Oct 29 22:01:54.871639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.871592 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" Oct 29 22:01:54.871639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.871602 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerDied","Data":"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed"} Oct 29 22:01:54.871891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.871659 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc" event={"ID":"d81945ba-078c-49ff-a4ef-60ec6ed2e21e","Type":"ContainerDied","Data":"3f80a2a3f4ddbcb45d1fa91c4dbdcbaaed67bbc79f89cd8a7a2d5844abb5ed50"} Oct 29 22:01:54.871891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.871686 4376 scope.go:122] "RemoveContainer" containerID="1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed" Oct 29 22:01:54.884318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.884300 4376 scope.go:122] "RemoveContainer" containerID="5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba" Oct 29 22:01:54.895350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.895331 4376 scope.go:122] "RemoveContainer" containerID="1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed" Oct 29 22:01:54.895632 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:01:54.895611 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed\": container with ID starting with 1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed not found: ID does not exist" containerID="1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed" Oct 29 22:01:54.895729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.895638 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed"} err="failed to get container status \"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed\": rpc error: code = NotFound desc = could not find container \"1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed\": container with ID starting with 1ed8c110fdd6f2af9e3d1e7e6a4c228b074f2395a8043f72596a5843ddb450ed not found: ID does not exist" Oct 29 22:01:54.895729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.895655 4376 scope.go:122] "RemoveContainer" containerID="5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba" Oct 29 22:01:54.895957 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:01:54.895930 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba\": container with ID starting with 5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba not found: ID does not exist" containerID="5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba" Oct 29 22:01:54.896009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.895966 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba"} err="failed to get container status \"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba\": rpc error: code = NotFound desc = could not find container \"5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba\": container with ID starting with 5cae2ddc2aa49b66778a9f9c976ea550dcf21da98918b87a4ffa06030261c7ba not found: ID does not exist" Oct 29 22:01:54.961013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.960938 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 22:01:54.965272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:54.965247 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a116a2-openshift-default-75c56bfdd9-824mc"] Oct 29 22:01:55.276665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:55.276624 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" path="/var/lib/kubelet/pods/d81945ba-078c-49ff-a4ef-60ec6ed2e21e/volumes" Oct 29 22:01:55.695715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:55.695631 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:01:55.695715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:55.695692 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:01:56.063332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.063294 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:01:56.064107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.064083 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:56.064107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.064104 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:56.064284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.064273 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:56.064335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.064287 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:56.068260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.068240 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.090306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.080179 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:01:56.093214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.093178 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-76c786-serving-cert\"" Oct 29 22:01:56.164358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.164323 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.164520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.164477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhfd\" (UniqueName: \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.164563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.164538 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.164749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.164719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.271935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.270327 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.271935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.270554 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhfd\" (UniqueName: \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.271935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.270664 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.271935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.270798 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.275986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.275192 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.317095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.301536 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.317857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.317828 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.338193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.330665 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhfd\" (UniqueName: \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\") pod \"maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.403526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.398185 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:56.699710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.699629 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:01:56.700717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.700694 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:01:56.767219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.767195 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:01:56.768135 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:01:56.768111 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbbcea9_28a9_4221_a46b_98edd17d27e9.slice/crio-4da53f38fba80cbf1fa1e2443491ad99d9b5353753e307a886af1d1ebbc44822 WatchSource:0}: Error finding container 4da53f38fba80cbf1fa1e2443491ad99d9b5353753e307a886af1d1ebbc44822: Status 404 returned error can't find the container with id 4da53f38fba80cbf1fa1e2443491ad99d9b5353753e307a886af1d1ebbc44822 Oct 29 22:01:56.890406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:56.890372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" event={"ID":"6cbbcea9-28a9-4221-a46b-98edd17d27e9","Type":"ContainerStarted","Data":"4da53f38fba80cbf1fa1e2443491ad99d9b5353753e307a886af1d1ebbc44822"} Oct 29 22:01:57.898942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:57.898835 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" event={"ID":"6cbbcea9-28a9-4221-a46b-98edd17d27e9","Type":"ContainerStarted","Data":"c015161bc54f7fe38a97997088796b0c7b05f0c38e5a00a86c645efe9e9f1496"} Oct 29 22:01:57.898942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:57.898930 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:01:57.946022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:57.945967 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" podStartSLOduration=1.507999512 podStartE2EDuration="1.945950641s" totalImagesPullingTime="437.951129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:57.944804327 +0000 UTC m=+1176.793281172" watchObservedRunningTime="2026-10-29 22:01:57.945950641 +0000 UTC m=+1176.794427497" Oct 29 22:01:58.519745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.519712 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7776-00da99"] Oct 29 22:01:58.520440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.520425 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:58.520498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.520441 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81945ba-078c-49ff-a4ef-60ec6ed2e21e" containerName="istio-proxy" Oct 29 22:01:58.523391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.523360 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:01:58.545714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.545676 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-00da99"] Oct 29 22:01:58.549027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.549000 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:01:58.691013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.690966 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtff\" (UniqueName: \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\") pod \"test-curl-7776-00da99\" (UID: \"02c4fb88-100a-435f-a9f6-e87a79a43069\") " pod="opendatahub/test-curl-7776-00da99" Oct 29 22:01:58.793333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.793255 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtff\" (UniqueName: \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\") pod \"test-curl-7776-00da99\" (UID: \"02c4fb88-100a-435f-a9f6-e87a79a43069\") " pod="opendatahub/test-curl-7776-00da99" Oct 29 22:01:58.819168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.819140 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtff\" (UniqueName: \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\") pod \"test-curl-7776-00da99\" (UID: \"02c4fb88-100a-435f-a9f6-e87a79a43069\") " pod="opendatahub/test-curl-7776-00da99" Oct 29 22:01:58.836229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:58.836205 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:01:59.143732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:59.143701 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-00da99"] Oct 29 22:01:59.144177 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:01:59.144083 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c4fb88_100a_435f_a9f6_e87a79a43069.slice/crio-a84597ceef8bb327a80a771723bbd776c307e123e78290d26fe6f81c740163ee WatchSource:0}: Error finding container a84597ceef8bb327a80a771723bbd776c307e123e78290d26fe6f81c740163ee: Status 404 returned error can't find the container with id a84597ceef8bb327a80a771723bbd776c307e123e78290d26fe6f81c740163ee Oct 29 22:01:59.918599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:01:59.918556 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-00da99" event={"ID":"02c4fb88-100a-435f-a9f6-e87a79a43069","Type":"ContainerStarted","Data":"a84597ceef8bb327a80a771723bbd776c307e123e78290d26fe6f81c740163ee"} Oct 29 22:02:02.974582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:02.974542 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-00da99" event={"ID":"02c4fb88-100a-435f-a9f6-e87a79a43069","Type":"ContainerStarted","Data":"e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16"} Oct 29 22:02:03.021913 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:03.021790 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7776-00da99" podStartSLOduration=2.262524272 podStartE2EDuration="5.021772015s" totalImagesPullingTime="2.759247743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:03.019839855 +0000 UTC m=+1181.868316702" watchObservedRunningTime="2026-10-29 22:02:03.021772015 +0000 UTC m=+1181.870248861" Oct 29 22:02:03.914525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:03.914496 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:02:04.032893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:04.032857 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7776-00da99"] Oct 29 22:02:04.049025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:04.048995 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7776-00da99"] Oct 29 22:02:04.991036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:04.990979 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7776-00da99" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" containerName="test-curl-7776-00da99" containerID="cri-o://e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16" gracePeriod=2 Oct 29 22:02:05.006094 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:05.006062 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7776-00da99\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:02:07.286298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:07.286260 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:02:07.386142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:07.386113 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\" (UniqueName: \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\") pod \"02c4fb88-100a-435f-a9f6-e87a79a43069\" (UID: \"02c4fb88-100a-435f-a9f6-e87a79a43069\") " Oct 29 22:02:07.388096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:07.388070 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff" pod "02c4fb88-100a-435f-a9f6-e87a79a43069" (UID: "02c4fb88-100a-435f-a9f6-e87a79a43069"). InnerVolumeSpecName "kube-api-access-jmtff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:07.488367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:07.488335 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmtff\" (UniqueName: \"kubernetes.io/projected/02c4fb88-100a-435f-a9f6-e87a79a43069-kube-api-access-jmtff\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:08.014659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.014620 4376 generic.go:356] "Generic (PLEG): container finished" podID="02c4fb88-100a-435f-a9f6-e87a79a43069" containerID="e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16" exitCode=137 Oct 29 22:02:08.014659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.014668 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:02:08.014898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.014725 4376 scope.go:122] "RemoveContainer" containerID="e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16" Oct 29 22:02:08.025530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.025510 4376 scope.go:122] "RemoveContainer" containerID="e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16" Oct 29 22:02:08.025872 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:08.025850 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16\": container with ID starting with e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16 not found: ID does not exist" containerID="e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16" Oct 29 22:02:08.025984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.025883 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16"} err="failed to get container status \"e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16\": rpc error: code = NotFound desc = could not find container \"e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16\": container with ID starting with e7d179e6feaa19c5814951c32da8da70984db249609dda0d7213862a4804cc16 not found: ID does not exist" Oct 29 22:02:08.039708 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:08.039663 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7776-00da99\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:02:08.916102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.916067 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:02:08.917066 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.917033 4376 cpu_manager.go:414] "containerMap: removing container" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" containerName="test-curl-7776-00da99" Oct 29 22:02:08.917066 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.917057 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" containerName="test-curl-7776-00da99" Oct 29 22:02:08.917252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.917237 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" containerName="test-curl-7776-00da99" Oct 29 22:02:08.921033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.921010 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:08.936428 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:08.936364 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7776-00da99\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" pod="opendatahub/test-curl-7776-00da99" Oct 29 22:02:08.946710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.946676 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:02:08.947796 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:08.947772 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fcce6b-serving-cert\"" Oct 29 22:02:09.012298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.012261 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqct\" (UniqueName: \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.012518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.012315 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.012667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.012642 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.012724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.012683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.114645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.114610 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqct\" (UniqueName: \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.114819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.114671 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.115041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.115022 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.115095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.115056 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.115648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.115626 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.117224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.117204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.117328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.117304 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.140678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.140650 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqct\" (UniqueName: \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\") pod \"maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.235744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.235706 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:09.304829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.297260 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c4fb88-100a-435f-a9f6-e87a79a43069" path="/var/lib/kubelet/pods/02c4fb88-100a-435f-a9f6-e87a79a43069/volumes" Oct 29 22:02:09.631702 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:09.631671 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7665687b_bde5_4c7a_a654_5ed89dd89ca2.slice/crio-ef9ea42e33caeaba62856b5a1684c4195c48a117d28d1b0f6aff263774208c1c WatchSource:0}: Error finding container ef9ea42e33caeaba62856b5a1684c4195c48a117d28d1b0f6aff263774208c1c: Status 404 returned error can't find the container with id ef9ea42e33caeaba62856b5a1684c4195c48a117d28d1b0f6aff263774208c1c Oct 29 22:02:09.631979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:09.631951 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:02:10.030339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:10.030306 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" event={"ID":"7665687b-bde5-4c7a-a654-5ed89dd89ca2","Type":"ContainerStarted","Data":"ef9ea42e33caeaba62856b5a1684c4195c48a117d28d1b0f6aff263774208c1c"} Oct 29 22:02:11.040874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:11.040826 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" event={"ID":"7665687b-bde5-4c7a-a654-5ed89dd89ca2","Type":"ContainerStarted","Data":"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc"} Oct 29 22:02:11.041368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:11.040913 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:11.093557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:11.093503 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" podStartSLOduration=2.710062917 podStartE2EDuration="3.093486306s" totalImagesPullingTime="383.423389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:11.09157068 +0000 UTC m=+1189.940047521" watchObservedRunningTime="2026-10-29 22:02:11.093486306 +0000 UTC m=+1189.941963152" Oct 29 22:02:15.200834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:15.200798 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 22:02:16.507967 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:16.507931 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:02:17.053641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:17.053610 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:02:20.210998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:20.202970 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" containerID="cri-o://b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640" gracePeriod=25 Oct 29 22:02:22.686055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:22.685090 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:22.815930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:22.768984 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:22.840848 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:22.840809 4376 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.576s" Oct 29 22:02:22.964185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:22.961188 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.072512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.072472 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.143199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.143155 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.279176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.279051 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.359513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.359458 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.400108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.400069 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:02:23.698842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.698706 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.701639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.701440 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:02:23.770270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.770224 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.770897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.770870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:02:23.838729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:23.838677 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.060930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.036977 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:02:24.083930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.061506 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:02:24.177934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.122546 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.177934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.123627 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:02:24.249227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.249179 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.334140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.321272 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.334140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.321636 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:02:24.377038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.367114 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.377038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.367481 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:02:24.484936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.452231 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.484936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.477047 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:02:24.586982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.583258 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:02:24.712931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.706558 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:24.848068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.770615 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:02:24.920511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:24.920474 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:02:25.060928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.056684 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:02:25.158126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.130033 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:25.336925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.332440 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:25.402935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.393124 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:25.710123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.710073 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:25.792926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.784507 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:25.808703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.796735 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:02:25.854149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.853173 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:02:25.951220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.951183 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:02:25.959124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.959099 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:02:25.975972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.954647 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:25.985552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.985526 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d9f8c0-openshift-default-dockercfg-2zr65\"" Oct 29 22:02:25.996712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.996681 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:02:25.997089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.997058 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:02:25.997286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.997264 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:02:25.997670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:25.997650 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:02:26.034206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.034162 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.034523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.034499 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.034674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.034654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.034826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.034807 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.035004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.034986 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksbj\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.035177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.035138 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.035408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.035388 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.035629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.035611 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.035820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.035801 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.138445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.138403 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.139246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.139221 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.139941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.139919 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.141568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.141547 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.142423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.142402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.142950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.142930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.144076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.144057 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.144581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.144563 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksbj\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.145044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.145021 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.145952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.143976 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.146095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.139096 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.146203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.144466 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.146304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.142840 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.146416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.140600 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.172010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.171893 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.175574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.175535 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksbj\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.183767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.183743 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 22:02:26.184030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.183983 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" podUID="8467d658-5ac1-49de-ba22-d4dde48f1c57" containerName="maas-api" containerID="cri-o://10b301f17bdc01ee673bc45e74d74c452e2133898d219737a16704f1383debeb" gracePeriod=30 Oct 29 22:02:26.187603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.186619 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.187696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.187596 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\") pod \"e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.305517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.305488 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:26.489595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.482523 4376 generic.go:356] "Generic (PLEG): container finished" podID="8467d658-5ac1-49de-ba22-d4dde48f1c57" containerID="10b301f17bdc01ee673bc45e74d74c452e2133898d219737a16704f1383debeb" exitCode=0 Oct 29 22:02:26.490748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:26.489926 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" event={"ID":"8467d658-5ac1-49de-ba22-d4dde48f1c57","Type":"ContainerDied","Data":"10b301f17bdc01ee673bc45e74d74c452e2133898d219737a16704f1383debeb"} Oct 29 22:02:27.372147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.365922 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:02:27.600976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.583248 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" event={"ID":"8467d658-5ac1-49de-ba22-d4dde48f1c57","Type":"ContainerDied","Data":"b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5"} Oct 29 22:02:27.600976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.583311 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b801ef90b237492e09b06bc9fae787a365a63d9bb139b2d89eebdc73270127a5" Oct 29 22:02:27.600976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.583432 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:27.602642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.601947 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:27.602642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.602030 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:27.602642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.602060 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:27.621620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.621218 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 22:02:27.701749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.701079 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\") pod \"8467d658-5ac1-49de-ba22-d4dde48f1c57\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " Oct 29 22:02:27.702647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.702623 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\") pod \"8467d658-5ac1-49de-ba22-d4dde48f1c57\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " Oct 29 22:02:27.704437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.704415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\") pod \"8467d658-5ac1-49de-ba22-d4dde48f1c57\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " Oct 29 22:02:27.704936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.702343 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca" pod "8467d658-5ac1-49de-ba22-d4dde48f1c57" (UID: "8467d658-5ac1-49de-ba22-d4dde48f1c57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:27.705208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.705188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\" (UniqueName: \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\") pod \"8467d658-5ac1-49de-ba22-d4dde48f1c57\" (UID: \"8467d658-5ac1-49de-ba22-d4dde48f1c57\") " Oct 29 22:02:27.706601 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.706578 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8467d658-5ac1-49de-ba22-d4dde48f1c57-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.734577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.734541 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99" pod "8467d658-5ac1-49de-ba22-d4dde48f1c57" (UID: "8467d658-5ac1-49de-ba22-d4dde48f1c57"). InnerVolumeSpecName "kube-api-access-jlh99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:27.742119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.742069 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs" pod "8467d658-5ac1-49de-ba22-d4dde48f1c57" (UID: "8467d658-5ac1-49de-ba22-d4dde48f1c57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:27.743160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.742947 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls" pod "8467d658-5ac1-49de-ba22-d4dde48f1c57" (UID: "8467d658-5ac1-49de-ba22-d4dde48f1c57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:27.813123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.810366 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.813123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.810415 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8467d658-5ac1-49de-ba22-d4dde48f1c57-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.813123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:27.810430 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlh99\" (UniqueName: \"kubernetes.io/projected/8467d658-5ac1-49de-ba22-d4dde48f1c57-kube-api-access-jlh99\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:28.593623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.593587 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerStarted","Data":"e7ca71e58a9d5c412bc781f96652d426fecea53fe809879673e8ee1005ec71be"} Oct 29 22:02:28.594120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.593648 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerStarted","Data":"7025e8dbb708ec1a8d2e3d806267f6a60d7b6cddb5a139d8369d842c2892c4a6"} Oct 29 22:02:28.594120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.593667 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4" Oct 29 22:02:28.648113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.648062 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podStartSLOduration=4.648048249 podStartE2EDuration="4.648048249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:28.645774823 +0000 UTC m=+1207.494251683" watchObservedRunningTime="2026-10-29 22:02:28.648048249 +0000 UTC m=+1207.496525094" Oct 29 22:02:28.688663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.688618 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 22:02:28.695157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:28.695132 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4082f-6f59c84dff-gmmm4"] Oct 29 22:02:29.273958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:29.273925 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8467d658-5ac1-49de-ba22-d4dde48f1c57" path="/var/lib/kubelet/pods/8467d658-5ac1-49de-ba22-d4dde48f1c57/volumes" Oct 29 22:02:29.306566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:29.306535 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:29.312540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:29.312510 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:29.611720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:29.611641 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:29.625002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:29.619035 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:02:33.181303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.181229 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:02:33.182062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.182043 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8467d658-5ac1-49de-ba22-d4dde48f1c57" containerName="maas-api" Oct 29 22:02:33.182062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.182063 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8467d658-5ac1-49de-ba22-d4dde48f1c57" containerName="maas-api" Oct 29 22:02:33.182229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.182220 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8467d658-5ac1-49de-ba22-d4dde48f1c57" containerName="maas-api" Oct 29 22:02:33.184894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.184876 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.205010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.204948 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:02:33.214080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.214055 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d9f8c0\"" Oct 29 22:02:33.214230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.214154 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d9f8c0-dockercfg-6bpbs\"" Oct 29 22:02:33.399308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.399266 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x699f\" (UniqueName: \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.399764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.399738 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.595847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.595757 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x699f\" (UniqueName: \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.596687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.596653 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.597886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.597862 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.663036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.643563 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:02:33.663036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.646499 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x699f\" (UniqueName: \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.701326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.701278 4376 generic.go:356] "Generic (PLEG): container finished" podID="cc336c69-f397-4df8-b330-d2945537d933" containerID="d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872" exitCode=0 Oct 29 22:02:33.702767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.702728 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerDied","Data":"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872"} Oct 29 22:02:33.705054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.705020 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:02:33.705263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.703000 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.705716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.705686 4376 scope.go:122] "RemoveContainer" containerID="d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872" Oct 29 22:02:33.708514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.708480 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:33.709198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.709170 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:33.709366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.709344 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:33.799744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.799711 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:33.800596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.800569 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7xv\" (UniqueName: \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.801790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.801761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.906657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.905038 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7xv\" (UniqueName: \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.906657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.905187 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.906657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.906273 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:33.934571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:33.934406 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7xv\" (UniqueName: \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\") pod \"payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:34.013427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.013383 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:02:34.026697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.026654 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:34.172475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.171601 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:02:34.333380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.333353 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:02:34.334066 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:34.334036 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5778348_6329_49f9_b2a5_6c9465a2c7e1.slice/crio-4f0841f04f0045f2f8168ec7fc60ac5a49b77e5ffc4fb6923c3ced7ad4d34975 WatchSource:0}: Error finding container 4f0841f04f0045f2f8168ec7fc60ac5a49b77e5ffc4fb6923c3ced7ad4d34975: Status 404 returned error can't find the container with id 4f0841f04f0045f2f8168ec7fc60ac5a49b77e5ffc4fb6923c3ced7ad4d34975 Oct 29 22:02:34.580235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.580198 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:02:34.580463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.580430 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" podUID="82fe5a85-b397-4875-8003-9c14f662c02b" containerName="maas-api" containerID="cri-o://c11593273ec5a29f5c74572b96f02c0610f6a22d97626de582cca1ba867e8960" gracePeriod=30 Oct 29 22:02:34.712242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.712207 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerStarted","Data":"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4"} Oct 29 22:02:34.714096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.714065 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" event={"ID":"abc230b1-37be-4146-b032-196373c93cb9","Type":"ContainerStarted","Data":"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180"} Oct 29 22:02:34.714288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.714108 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" event={"ID":"abc230b1-37be-4146-b032-196373c93cb9","Type":"ContainerStarted","Data":"b318c0186049d68e93175b64a577d871da5f479b45cdd74cebf218ae7967a771"} Oct 29 22:02:34.714288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.714158 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:34.716031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.716003 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" event={"ID":"f5778348-6329-49f9-b2a5-6c9465a2c7e1","Type":"ContainerStarted","Data":"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150"} Oct 29 22:02:34.716127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.716052 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" event={"ID":"f5778348-6329-49f9-b2a5-6c9465a2c7e1","Type":"ContainerStarted","Data":"4f0841f04f0045f2f8168ec7fc60ac5a49b77e5ffc4fb6923c3ced7ad4d34975"} Oct 29 22:02:34.716127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.716077 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:34.717635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.717615 4376 generic.go:356] "Generic (PLEG): container finished" podID="82fe5a85-b397-4875-8003-9c14f662c02b" containerID="c11593273ec5a29f5c74572b96f02c0610f6a22d97626de582cca1ba867e8960" exitCode=0 Oct 29 22:02:34.717732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.717695 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" event={"ID":"82fe5a85-b397-4875-8003-9c14f662c02b","Type":"ContainerDied","Data":"c11593273ec5a29f5c74572b96f02c0610f6a22d97626de582cca1ba867e8960"} Oct 29 22:02:34.810985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.810924 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" podStartSLOduration=1.810885199 podStartE2EDuration="1.810885199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:34.810025704 +0000 UTC m=+1213.658502549" watchObservedRunningTime="2026-10-29 22:02:34.810885199 +0000 UTC m=+1213.659362045" Oct 29 22:02:34.868684 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.868365 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" podStartSLOduration=1.868345119 podStartE2EDuration="1.868345119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:34.865322828 +0000 UTC m=+1213.713799677" watchObservedRunningTime="2026-10-29 22:02:34.868345119 +0000 UTC m=+1213.716821967" Oct 29 22:02:34.879658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.879626 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:02:34.903369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.903334 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:02:34.904606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.904585 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:34.904670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.904628 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:34.926689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.926657 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\") pod \"82fe5a85-b397-4875-8003-9c14f662c02b\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " Oct 29 22:02:34.929945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.929915 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") pod \"82fe5a85-b397-4875-8003-9c14f662c02b\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " Oct 29 22:02:34.930188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.930159 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\") pod \"82fe5a85-b397-4875-8003-9c14f662c02b\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " Oct 29 22:02:34.930332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.930249 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\" (UniqueName: \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\") pod \"82fe5a85-b397-4875-8003-9c14f662c02b\" (UID: \"82fe5a85-b397-4875-8003-9c14f662c02b\") " Oct 29 22:02:34.949592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.939385 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls" pod "82fe5a85-b397-4875-8003-9c14f662c02b" (UID: "82fe5a85-b397-4875-8003-9c14f662c02b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:34.949592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.941274 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca" pod "82fe5a85-b397-4875-8003-9c14f662c02b" (UID: "82fe5a85-b397-4875-8003-9c14f662c02b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:34.982534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.982480 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6" pod "82fe5a85-b397-4875-8003-9c14f662c02b" (UID: "82fe5a85-b397-4875-8003-9c14f662c02b"). InnerVolumeSpecName "kube-api-access-6hzp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:34.984958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.984882 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs" pod "82fe5a85-b397-4875-8003-9c14f662c02b" (UID: "82fe5a85-b397-4875-8003-9c14f662c02b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:34.994583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.994559 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.994668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.994589 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fe5a85-b397-4875-8003-9c14f662c02b-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.994668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.994604 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzp6\" (UniqueName: \"kubernetes.io/projected/82fe5a85-b397-4875-8003-9c14f662c02b-kube-api-access-6hzp6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.994668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:34.994626 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fe5a85-b397-4875-8003-9c14f662c02b-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:35.729132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.729092 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" event={"ID":"82fe5a85-b397-4875-8003-9c14f662c02b","Type":"ContainerDied","Data":"b8191ccbf3ef67af98aab6e872fc19caa09cfdcf343ad17b16c65e2ccf24a3b5"} Oct 29 22:02:35.729587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.729164 4376 scope.go:122] "RemoveContainer" containerID="c11593273ec5a29f5c74572b96f02c0610f6a22d97626de582cca1ba867e8960" Oct 29 22:02:35.729587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.729294 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb" Oct 29 22:02:35.808887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.808853 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:02:35.821678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.821651 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7f9e0-69fc4b865c-4q4rb"] Oct 29 22:02:35.904119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.904087 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:35.904283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:35.904136 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:36.055979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:36.054986 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:02:36.574033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:36.573385 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 22:02:36.574033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:36.573632 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" containerID="cri-o://6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47" gracePeriod=30 Oct 29 22:02:36.909979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:36.905135 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:36.909979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:36.905244 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:37.280744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.280705 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fe5a85-b397-4875-8003-9c14f662c02b" path="/var/lib/kubelet/pods/82fe5a85-b397-4875-8003-9c14f662c02b/volumes" Oct 29 22:02:37.281160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.281139 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:02:37.281944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.281888 4376 cpu_manager.go:414] "containerMap: removing container" podUID="82fe5a85-b397-4875-8003-9c14f662c02b" containerName="maas-api" Oct 29 22:02:37.281944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.281924 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fe5a85-b397-4875-8003-9c14f662c02b" containerName="maas-api" Oct 29 22:02:37.282172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.282155 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fe5a85-b397-4875-8003-9c14f662c02b" containerName="maas-api" Oct 29 22:02:37.291516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.291492 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:02:37.291972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.291947 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.323839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.323816 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-296d19-openshift-default-dockercfg-mmgd6\"" Oct 29 22:02:37.363305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.363267 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.363513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.363339 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.363513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.363458 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.363982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.363887 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.364111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.364035 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.364111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.364072 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.364111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.364092 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.364293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.364171 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.364293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.364188 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xz4\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.465834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.465796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.465854 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.465983 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466035 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466184 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466226 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466292 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466335 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xz4\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466390 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466585 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.466971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.466941 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.468556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.468531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.469032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.469007 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.513938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.499857 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xz4\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.513938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.502306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\") pod \"e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.631988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.631702 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:37.963828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.948957 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:37.963828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:37.949025 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:38.296604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.296570 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:02:38.296825 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:38.296800 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf57fbd_1cca_4e9d_b3fe_05b24a88a6dd.slice/crio-908ace16ad3a645198662246633c10258745c4aaa1c3b1b5d6eb226356c481ca WatchSource:0}: Error finding container 908ace16ad3a645198662246633c10258745c4aaa1c3b1b5d6eb226356c481ca: Status 404 returned error can't find the container with id 908ace16ad3a645198662246633c10258745c4aaa1c3b1b5d6eb226356c481ca Oct 29 22:02:38.299576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.299539 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:38.299666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.299608 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:38.299666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.299642 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:38.869699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.869666 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerStarted","Data":"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984"} Oct 29 22:02:38.869877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.869723 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerStarted","Data":"908ace16ad3a645198662246633c10258745c4aaa1c3b1b5d6eb226356c481ca"} Oct 29 22:02:38.904054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.904025 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:38.904228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.904071 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:38.923503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:38.923442 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podStartSLOduration=1.9234241650000001 podStartE2EDuration="1.923424165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:38.918592244 +0000 UTC m=+1217.767069091" watchObservedRunningTime="2026-10-29 22:02:38.923424165 +0000 UTC m=+1217.771901014" Oct 29 22:02:39.632302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:39.632265 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:39.844929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:39.838933 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:39.903469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:39.903379 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:39.903723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:39.903690 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:40.010940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:39.991483 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:40.082958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.081643 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:02:40.576965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.576638 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 22:02:40.669808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.669774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\") pod \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " Oct 29 22:02:40.670236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.670041 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\" (UniqueName: \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\") pod \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\" (UID: \"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af\") " Oct 29 22:02:40.670279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.670230 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume" pod "03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" (UID: "03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:40.670804 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.670777 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:40.672581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.672551 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn" pod "03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" (UID: "03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af"). InnerVolumeSpecName "kube-api-access-ctcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:40.772204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.772119 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctcxn\" (UniqueName: \"kubernetes.io/projected/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af-kube-api-access-ctcxn\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:40.903518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.903484 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:02:40.903699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.903543 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:02:40.997160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.997116 4376 generic.go:356] "Generic (PLEG): container finished" podID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerID="b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640" exitCode=0 Oct 29 22:02:40.997345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.997169 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" event={"ID":"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af","Type":"ContainerDied","Data":"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640"} Oct 29 22:02:40.997345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.997200 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" Oct 29 22:02:40.997345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.997236 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8" event={"ID":"03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af","Type":"ContainerDied","Data":"24445e108b825c7e277351b6e958579febcbdcaefaca3d92f12ec3ea21051b8e"} Oct 29 22:02:40.997345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:40.997267 4376 scope.go:122] "RemoveContainer" containerID="b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640" Oct 29 22:02:41.014862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.014834 4376 scope.go:122] "RemoveContainer" containerID="b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640" Oct 29 22:02:41.015362 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:41.015329 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640\": container with ID starting with b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640 not found: ID does not exist" containerID="b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640" Oct 29 22:02:41.015473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.015361 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640"} err="failed to get container status \"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640\": rpc error: code = NotFound desc = could not find container \"b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640\": container with ID starting with b6948c1e6ad17a2e365f6473c352f8de5e489329509ab9d9c31e11802d46a640 not found: ID does not exist" Oct 29 22:02:41.064676 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.064579 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 22:02:41.069605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.069573 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-cfnd8"] Oct 29 22:02:41.274683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.274650 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" path="/var/lib/kubelet/pods/03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af/volumes" Oct 29 22:02:41.702231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.702196 4376 patch_prober.go:29] interesting pod/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:02:41.702581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.702256 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 29 22:02:41.871355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.871275 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:02:41.904875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.904843 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.904928 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.904953 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905115 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905206 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:41.905289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905273 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905407 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905336 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905407 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905370 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905515 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:41.905823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905591 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.905823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:41.905823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905676 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\") pod \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\" (UID: \"d94dc4f0-3ea4-414c-b550-eb686cbf38ec\") " Oct 29 22:02:41.906115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905850 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:41.906115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.905864 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:41.906658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.906630 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.906770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.906666 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.906770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.906685 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.906770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.906700 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.906770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.906713 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.907777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.907751 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:02:41.907777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.907767 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "kube-api-access-dt2lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:41.907948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.907927 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:41.908276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.908243 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy" pod "d94dc4f0-3ea4-414c-b550-eb686cbf38ec" (UID: "d94dc4f0-3ea4-414c-b550-eb686cbf38ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:41.909600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.909580 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:02:41.910924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:41.910892 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:02:42.008551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.008516 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2lw\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-kube-api-access-dt2lw\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:42.008750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.008569 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:42.008750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.008587 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:42.008750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.008603 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94dc4f0-3ea4-414c-b550-eb686cbf38ec-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:42.011457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.011422 4376 generic.go:356] "Generic (PLEG): container finished" podID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerID="6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47" exitCode=0 Oct 29 22:02:42.011597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.011467 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" Oct 29 22:02:42.011597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.011500 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerDied","Data":"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47"} Oct 29 22:02:42.011597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.011562 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt" event={"ID":"d94dc4f0-3ea4-414c-b550-eb686cbf38ec","Type":"ContainerDied","Data":"3ceb4cdde67fc6b9bb4b18a3d7e2f419598e25b3aa67e27b5ff584ab7625b407"} Oct 29 22:02:42.011597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.011592 4376 scope.go:122] "RemoveContainer" containerID="6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47" Oct 29 22:02:42.024827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.024802 4376 scope.go:122] "RemoveContainer" containerID="e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24" Oct 29 22:02:42.035451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.035426 4376 scope.go:122] "RemoveContainer" containerID="6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47" Oct 29 22:02:42.035841 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:42.035815 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47\": container with ID starting with 6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47 not found: ID does not exist" containerID="6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47" Oct 29 22:02:42.035951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.035852 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47"} err="failed to get container status \"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47\": rpc error: code = NotFound desc = could not find container \"6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47\": container with ID starting with 6a6025999a61893bd9c211f1863dd1b057084341c2dfcbad4b395b3cf56c8f47 not found: ID does not exist" Oct 29 22:02:42.035951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.035871 4376 scope.go:122] "RemoveContainer" containerID="e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24" Oct 29 22:02:42.036201 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:42.036177 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24\": container with ID starting with e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24 not found: ID does not exist" containerID="e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24" Oct 29 22:02:42.036245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.036207 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24"} err="failed to get container status \"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24\": rpc error: code = NotFound desc = could not find container \"e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24\": container with ID starting with e776c7cff4a880444d229288b3d11d25f8b060dfc2f204016c6b7f1f1f302c24 not found: ID does not exist" Oct 29 22:02:42.090200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.090160 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 22:02:42.095139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:42.095104 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4082f-openshift-default-75db75cddf-rbtrt"] Oct 29 22:02:43.273757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.273724 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" path="/var/lib/kubelet/pods/d94dc4f0-3ea4-414c-b550-eb686cbf38ec/volumes" Oct 29 22:02:43.348599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.348568 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:02:43.349364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349346 4376 cpu_manager.go:414] "containerMap: removing container" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" Oct 29 22:02:43.349408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349366 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" Oct 29 22:02:43.349468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349419 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349424 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349438 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349444 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349538 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349547 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94dc4f0-3ea4-414c-b550-eb686cbf38ec" containerName="istio-proxy" Oct 29 22:02:43.349592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.349554 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c7dc9b-11b6-47a1-b8d8-c76a71b0c3af" containerName="payload-processing" Oct 29 22:02:43.353274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.353255 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:43.375038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.375014 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:02:43.382782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.382759 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-296d19-dockercfg-8vzmd\"" Oct 29 22:02:43.382955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.382774 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-296d19\"" Oct 29 22:02:43.544979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.514713 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wzq\" (UniqueName: \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:43.666975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.639473 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:02:43.721946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.690976 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:43.726524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.698763 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.848402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wzq\" (UniqueName: \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.848561 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68z4\" (UniqueName: \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.848695 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.848825 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.854529 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:02:43.876635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:43.855577 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:44.084974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.078787 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68z4\" (UniqueName: \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:44.084974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.079182 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:44.084974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.080835 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:44.153505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.152623 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68z4\" (UniqueName: \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\") pod \"payload-processing-e2e-api-b-296d19-7747cc4c98-495l6\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:44.162787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.162755 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wzq\" (UniqueName: \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\") pod \"payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:44.275017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.259421 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:44.276013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.275695 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:44.828040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.828004 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:02:44.828612 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:44.828585 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660565e7_f6c3_4b24_939e_b7ad7f095b6b.slice/crio-d0212c5db9b20c4ee76f92fb4ef14d263c6dead62bafab74e37f11345a01af39 WatchSource:0}: Error finding container d0212c5db9b20c4ee76f92fb4ef14d263c6dead62bafab74e37f11345a01af39: Status 404 returned error can't find the container with id d0212c5db9b20c4ee76f92fb4ef14d263c6dead62bafab74e37f11345a01af39 Oct 29 22:02:44.863778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:44.863742 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:02:44.866358 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:44.866330 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c741a1d_737c_4568_bc28_69b6a434f51f.slice/crio-04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b WatchSource:0}: Error finding container 04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b: Status 404 returned error can't find the container with id 04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b Oct 29 22:02:45.341277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.341233 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" event={"ID":"8c741a1d-737c-4568-bc28-69b6a434f51f","Type":"ContainerStarted","Data":"846b8516867e03f812835a97f2483c24fdefa8c69830b45de5c8f36542236dea"} Oct 29 22:02:45.341720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.341291 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" event={"ID":"8c741a1d-737c-4568-bc28-69b6a434f51f","Type":"ContainerStarted","Data":"04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b"} Oct 29 22:02:45.341720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.341336 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:45.343149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.343120 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" event={"ID":"660565e7-f6c3-4b24-939e-b7ad7f095b6b","Type":"ContainerStarted","Data":"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59"} Oct 29 22:02:45.343288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.343163 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" event={"ID":"660565e7-f6c3-4b24-939e-b7ad7f095b6b","Type":"ContainerStarted","Data":"d0212c5db9b20c4ee76f92fb4ef14d263c6dead62bafab74e37f11345a01af39"} Oct 29 22:02:45.343288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.343208 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:45.386690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.386630 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" podStartSLOduration=2.386612387 podStartE2EDuration="2.386612387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:45.386057153 +0000 UTC m=+1224.234533999" watchObservedRunningTime="2026-10-29 22:02:45.386612387 +0000 UTC m=+1224.235089236" Oct 29 22:02:45.432452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.432400 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" podStartSLOduration=2.432387323 podStartE2EDuration="2.432387323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:45.430540685 +0000 UTC m=+1224.279017532" watchObservedRunningTime="2026-10-29 22:02:45.432387323 +0000 UTC m=+1224.280864168" Oct 29 22:02:45.732396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.732372 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:02:45.732558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:45.732434 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:02:46.433989 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:02:46.433953 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:02:46.820939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.817643 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:02:46.884929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.884156 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:02:46.884929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.884291 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:46.981513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.981102 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76c786-dockercfg-9zp5h\"" Oct 29 22:02:46.981513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.981180 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76c786\"" Oct 29 22:02:46.990015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.988602 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:02:46.999015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:46.993288 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.023114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.020449 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.023114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.020548 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntf8w\" (UniqueName: \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.023114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.020645 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.023114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.020722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdtv\" (UniqueName: \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.129176 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf8w\" (UniqueName: \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.129298 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.129377 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdtv\" (UniqueName: \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.129650 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.130688 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.142100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.131807 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.167632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.167580 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:02:47.186542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.186508 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntf8w\" (UniqueName: \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\") pod \"payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.196367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.196309 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 22:02:47.196672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.196620 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" podUID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" containerName="maas-api" containerID="cri-o://59d0cb1c8ed031c875759f98fc6f486824a32ca7e7aaacbb18d77f17c060a789" gracePeriod=30 Oct 29 22:02:47.225991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.221743 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdtv\" (UniqueName: \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\") pod \"payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.258591 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.258296 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:47.354298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.354235 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:47.366291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.366252 4376 generic.go:356] "Generic (PLEG): container finished" podID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" containerID="59d0cb1c8ed031c875759f98fc6f486824a32ca7e7aaacbb18d77f17c060a789" exitCode=0 Oct 29 22:02:47.366421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.366401 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" event={"ID":"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f","Type":"ContainerDied","Data":"59d0cb1c8ed031c875759f98fc6f486824a32ca7e7aaacbb18d77f17c060a789"} Oct 29 22:02:47.913119 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:02:47.910179 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c1b211_4366_492e_af83_5b1fd097c8d9.slice/crio-dbee440b897c128691c713bf44b13e67a191101d537266f4641a95ea179c2601 WatchSource:0}: Error finding container dbee440b897c128691c713bf44b13e67a191101d537266f4641a95ea179c2601: Status 404 returned error can't find the container with id dbee440b897c128691c713bf44b13e67a191101d537266f4641a95ea179c2601 Oct 29 22:02:47.975959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:47.930393 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:02:48.228989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.226735 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.296188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\") pod \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.296402 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\") pod \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.296509 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\" (UniqueName: \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\") pod \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.296552 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\") pod \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\" (UID: \"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f\") " Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.296585 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca" pod "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" (UID: "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:48.297986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.297241 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:48.300602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.298992 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls" pod "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" (UID: "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:48.319936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.319814 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t" pod "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" (UID: "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f"). InnerVolumeSpecName "kube-api-access-pgn2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:48.320112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.319964 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs" pod "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" (UID: "8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:48.331388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.331344 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:02:48.374682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.374648 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" event={"ID":"8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f","Type":"ContainerDied","Data":"897132697fb6d90033023d17a3c132901aad767d7d4b727d0a0d3345c1756abf"} Oct 29 22:02:48.374819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.374664 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb" Oct 29 22:02:48.374819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.374728 4376 scope.go:122] "RemoveContainer" containerID="59d0cb1c8ed031c875759f98fc6f486824a32ca7e7aaacbb18d77f17c060a789" Oct 29 22:02:48.376306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.376277 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" event={"ID":"97489e7f-829c-4041-bf51-ef7ba61692c1","Type":"ContainerStarted","Data":"b67a427386d541eae80f14fafb9bac95b1615a523988c62dca6251b35f5500ec"} Oct 29 22:02:48.377752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.377730 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" event={"ID":"b8c1b211-4366-492e-af83-5b1fd097c8d9","Type":"ContainerStarted","Data":"dbee440b897c128691c713bf44b13e67a191101d537266f4641a95ea179c2601"} Oct 29 22:02:48.398753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.398727 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:48.398860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.398784 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgn2t\" (UniqueName: \"kubernetes.io/projected/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-kube-api-access-pgn2t\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:48.398860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.398799 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:48.437658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.437634 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 22:02:48.442779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:48.442757 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-ln9fb"] Oct 29 22:02:49.284521 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:49.284482 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" path="/var/lib/kubelet/pods/8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f/volumes" Oct 29 22:02:51.843544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.843504 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:02:51.870928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.851332 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" containerName="maas-api" Oct 29 22:02:51.913927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.906156 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" containerName="maas-api" Oct 29 22:02:51.913927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.906511 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef1f25f-072e-4edb-9e17-9f54c6ff5b8f" containerName="maas-api" Oct 29 22:02:52.022929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.997638 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:02:52.023220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:51.997926 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.047657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.047445 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fd295\"" Oct 29 22:02:52.061726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.061689 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fd295-dockercfg-w5nlp\"" Oct 29 22:02:52.100185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.100018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.100839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.100795 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5pd\" (UniqueName: \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.202795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.202766 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.203092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.202991 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5pd\" (UniqueName: \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.203562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.203537 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.229490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.229464 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5pd\" (UniqueName: \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\") pod \"payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.465171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.465137 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:52.662958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.658390 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" event={"ID":"97489e7f-829c-4041-bf51-ef7ba61692c1","Type":"ContainerStarted","Data":"639114cbfebcfcd1cf45d1126eeac327e0c47eea94ebd267b31c9933298de2e5"} Oct 29 22:02:52.662958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.659614 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:02:52.662958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.662488 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" event={"ID":"b8c1b211-4366-492e-af83-5b1fd097c8d9","Type":"ContainerStarted","Data":"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de"} Oct 29 22:02:52.665090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.663297 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:02:52.730207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.730132 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" podStartSLOduration=2.86875456 podStartE2EDuration="6.730112933s" totalImagesPullingTime="3.861358373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:52.718356138 +0000 UTC m=+1231.566832978" watchObservedRunningTime="2026-10-29 22:02:52.730112933 +0000 UTC m=+1231.578589780" Oct 29 22:02:52.977927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.958163 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" podStartSLOduration=2.708572075 podStartE2EDuration="6.958140483s" totalImagesPullingTime="4.249568408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:52.795718759 +0000 UTC m=+1231.644195608" watchObservedRunningTime="2026-10-29 22:02:52.958140483 +0000 UTC m=+1231.806617330" Oct 29 22:02:53.001094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:52.999637 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:02:53.683696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:53.683646 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" event={"ID":"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e","Type":"ContainerStarted","Data":"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815"} Oct 29 22:02:53.683953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:53.683710 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" event={"ID":"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e","Type":"ContainerStarted","Data":"bb7306beaa78dbdaaf842d010a85f1098f4a6103c9db4a1d8b4e7a022a3f3430"} Oct 29 22:02:53.684059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:53.684004 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:02:53.733805 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:53.733749 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" podStartSLOduration=2.733735589 podStartE2EDuration="2.733735589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:53.730386856 +0000 UTC m=+1232.578863705" watchObservedRunningTime="2026-10-29 22:02:53.733735589 +0000 UTC m=+1232.582212435" Oct 29 22:02:56.356570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:56.356541 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:02:56.357057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:56.356612 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:02:59.796993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.796939 4376 generic.go:356] "Generic (PLEG): container finished" podID="bf40a427-273f-4d62-96fc-e866a89e9726" containerID="da5113c9ecbf316d808f3a989b1427271ebf95f6de3e2c43019727f71a97d280" exitCode=0 Oct 29 22:02:59.797547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.797144 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerDied","Data":"da5113c9ecbf316d808f3a989b1427271ebf95f6de3e2c43019727f71a97d280"} Oct 29 22:02:59.797719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.797699 4376 scope.go:122] "RemoveContainer" containerID="da5113c9ecbf316d808f3a989b1427271ebf95f6de3e2c43019727f71a97d280" Oct 29 22:02:59.800301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.800264 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:59.800405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.800374 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:02:59.800452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:02:59.800414 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:01.134393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:01.134346 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerStarted","Data":"730c58e19f27891c2eb9d797f983d16619f2d2e1049b0f0d6c5ef3466a1f08c8"} Oct 29 22:03:01.594525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:01.594490 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:01.595620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:01.595593 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:01.595730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:01.595639 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:02.472830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:02.472790 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:02.594927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:02.594137 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:02.594927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:02.594214 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:03.598879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:03.598407 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:03.598879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:03.598472 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:04.598926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:04.595307 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:04.598926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:04.595362 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:04.699939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:04.699734 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:03:04.708933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:04.708045 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:03:04.712925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:04.712022 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:03:05.599346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:05.595289 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:05.599346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:05.595362 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:06.600353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:06.600313 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:03:06.600979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:06.600935 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:03:06.879842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:06.879752 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:03:06.940937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:06.936002 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:03:07.093399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:07.079351 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:03:07.599284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:07.599250 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:07.600652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:07.600622 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:14.755089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.755049 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:03:14.780930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.775477 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.798640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.798611 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8867e7d-serving-cert\"" Oct 29 22:03:14.818927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.815057 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:03:14.862288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.862246 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.862484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.862325 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.862484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.862368 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckhk\" (UniqueName: \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.862484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.862468 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.976278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.971315 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.976278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.971394 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.976278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.971430 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckhk\" (UniqueName: \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.976278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.971489 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:14.976278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:14.974856 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:15.026975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.014739 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckhk\" (UniqueName: \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:15.030926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.027624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:15.046928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.044775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\") pod \"maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:15.128983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.128944 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:15.466018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.465986 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:03:15.466148 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:03:15.466029 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda610ae81_2434_4735_b850_2818a82f478a.slice/crio-cec427f4ae60641a376f25cb6f364a74e62008a76f7a5454da59e7468160dbbc WatchSource:0}: Error finding container cec427f4ae60641a376f25cb6f364a74e62008a76f7a5454da59e7468160dbbc: Status 404 returned error can't find the container with id cec427f4ae60641a376f25cb6f364a74e62008a76f7a5454da59e7468160dbbc Oct 29 22:03:15.629920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:15.629862 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" event={"ID":"a610ae81-2434-4735-b850-2818a82f478a","Type":"ContainerStarted","Data":"cec427f4ae60641a376f25cb6f364a74e62008a76f7a5454da59e7468160dbbc"} Oct 29 22:03:16.740794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:16.740756 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" event={"ID":"a610ae81-2434-4735-b850-2818a82f478a","Type":"ContainerStarted","Data":"707eaba34a3a0b8b68f0747676fb56c8c6bdcb370e15c6602bf1a0122f8ab89d"} Oct 29 22:03:16.742756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:16.742732 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:16.833125 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:03:16.833082 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:03:16.835010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:16.834942 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" podStartSLOduration=2.441618876 podStartE2EDuration="2.834923353s" totalImagesPullingTime="393.304477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:16.828542255 +0000 UTC m=+1255.677019100" watchObservedRunningTime="2026-10-29 22:03:16.834923353 +0000 UTC m=+1255.683400194" Oct 29 22:03:23.817345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:23.817318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:03:26.225546 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.225511 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:03:26.226186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.225796 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" podUID="cb13f771-8a74-4195-871a-9a0a4fb1b454" containerName="maas-api" containerID="cri-o://2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d" gracePeriod=30 Oct 29 22:03:26.734570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.731288 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:03:26.768947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.767959 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\") pod \"cb13f771-8a74-4195-871a-9a0a4fb1b454\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " Oct 29 22:03:26.768947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.768261 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\") pod \"cb13f771-8a74-4195-871a-9a0a4fb1b454\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " Oct 29 22:03:26.768947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.768399 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\" (UniqueName: \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\") pod \"cb13f771-8a74-4195-871a-9a0a4fb1b454\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " Oct 29 22:03:26.768947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.768435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\") pod \"cb13f771-8a74-4195-871a-9a0a4fb1b454\" (UID: \"cb13f771-8a74-4195-871a-9a0a4fb1b454\") " Oct 29 22:03:26.795966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.792288 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca" pod "cb13f771-8a74-4195-871a-9a0a4fb1b454" (UID: "cb13f771-8a74-4195-871a-9a0a4fb1b454"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:26.802530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.802483 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls" pod "cb13f771-8a74-4195-871a-9a0a4fb1b454" (UID: "cb13f771-8a74-4195-871a-9a0a4fb1b454"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:03:26.803058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.803016 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs" pod "cb13f771-8a74-4195-871a-9a0a4fb1b454" (UID: "cb13f771-8a74-4195-871a-9a0a4fb1b454"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:03:26.808622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.808542 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z" pod "cb13f771-8a74-4195-871a-9a0a4fb1b454" (UID: "cb13f771-8a74-4195-871a-9a0a4fb1b454"). InnerVolumeSpecName "kube-api-access-8sb9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:26.945547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.945507 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb13f771-8a74-4195-871a-9a0a4fb1b454-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:26.977029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.976997 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sb9z\" (UniqueName: \"kubernetes.io/projected/cb13f771-8a74-4195-871a-9a0a4fb1b454-kube-api-access-8sb9z\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:26.977274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.977253 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:26.977397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:26.977384 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb13f771-8a74-4195-871a-9a0a4fb1b454-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:27.180378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.177342 4376 generic.go:356] "Generic (PLEG): container finished" podID="cb13f771-8a74-4195-871a-9a0a4fb1b454" containerID="2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d" exitCode=0 Oct 29 22:03:27.180378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.177716 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" event={"ID":"cb13f771-8a74-4195-871a-9a0a4fb1b454","Type":"ContainerDied","Data":"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d"} Oct 29 22:03:27.180378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.177784 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" event={"ID":"cb13f771-8a74-4195-871a-9a0a4fb1b454","Type":"ContainerDied","Data":"5e0fe25c7fcf5ee5e56a6215c1cd1ae4a39147a94ddb747542ea846e0702a7e1"} Oct 29 22:03:27.180378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.177816 4376 scope.go:122] "RemoveContainer" containerID="2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d" Oct 29 22:03:27.185035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.185005 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7" Oct 29 22:03:27.409173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.409132 4376 scope.go:122] "RemoveContainer" containerID="2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d" Oct 29 22:03:27.427934 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:03:27.422293 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d\": container with ID starting with 2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d not found: ID does not exist" containerID="2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d" Oct 29 22:03:27.427934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.422357 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d"} err="failed to get container status \"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d\": rpc error: code = NotFound desc = could not find container \"2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d\": container with ID starting with 2eb8980b82cc45eeeaa3f5f221356887602b56cbf0986b8d706a6aee8fbc7b5d not found: ID does not exist" Oct 29 22:03:27.474550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.474458 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:03:27.501178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.501117 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-579f09-755f848844-d8lk7"] Oct 29 22:03:27.507555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.506393 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:03:27.562377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:27.561632 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:03:29.298326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:29.298283 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb13f771-8a74-4195-871a-9a0a4fb1b454" path="/var/lib/kubelet/pods/cb13f771-8a74-4195-871a-9a0a4fb1b454/volumes" Oct 29 22:03:32.507188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:32.507149 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" podUID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" containerName="payload-processing" containerID="cri-o://4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150" gracePeriod=25 Oct 29 22:03:32.563411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:32.563354 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" podUID="abc230b1-37be-4146-b032-196373c93cb9" containerName="payload-pre-processing" containerID="cri-o://491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180" gracePeriod=25 Oct 29 22:03:32.987188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:32.987155 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:03:33.057683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.057646 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:03:33.139609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.139566 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\" (UniqueName: \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\") pod \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " Oct 29 22:03:33.139990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.139958 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\") pod \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\" (UID: \"f5778348-6329-49f9-b2a5-6c9465a2c7e1\") " Oct 29 22:03:33.140307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.140283 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume" pod "f5778348-6329-49f9-b2a5-6c9465a2c7e1" (UID: "f5778348-6329-49f9-b2a5-6c9465a2c7e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:33.140624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.140598 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5778348-6329-49f9-b2a5-6c9465a2c7e1-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:33.141762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.141740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv" pod "f5778348-6329-49f9-b2a5-6c9465a2c7e1" (UID: "f5778348-6329-49f9-b2a5-6c9465a2c7e1"). InnerVolumeSpecName "kube-api-access-zs7xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:33.220965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.220931 4376 generic.go:356] "Generic (PLEG): container finished" podID="abc230b1-37be-4146-b032-196373c93cb9" containerID="491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180" exitCode=0 Oct 29 22:03:33.220965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.220962 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" Oct 29 22:03:33.221224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.220997 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" event={"ID":"abc230b1-37be-4146-b032-196373c93cb9","Type":"ContainerDied","Data":"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180"} Oct 29 22:03:33.221224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.221080 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs" event={"ID":"abc230b1-37be-4146-b032-196373c93cb9","Type":"ContainerDied","Data":"b318c0186049d68e93175b64a577d871da5f479b45cdd74cebf218ae7967a771"} Oct 29 22:03:33.221224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.221110 4376 scope.go:122] "RemoveContainer" containerID="491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180" Oct 29 22:03:33.222460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.222441 4376 generic.go:356] "Generic (PLEG): container finished" podID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" containerID="4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150" exitCode=0 Oct 29 22:03:33.222539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.222470 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" event={"ID":"f5778348-6329-49f9-b2a5-6c9465a2c7e1","Type":"ContainerDied","Data":"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150"} Oct 29 22:03:33.222539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.222498 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" Oct 29 22:03:33.222539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.222518 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55" event={"ID":"f5778348-6329-49f9-b2a5-6c9465a2c7e1","Type":"ContainerDied","Data":"4f0841f04f0045f2f8168ec7fc60ac5a49b77e5ffc4fb6923c3ced7ad4d34975"} Oct 29 22:03:33.232465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.232447 4376 scope.go:122] "RemoveContainer" containerID="491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180" Oct 29 22:03:33.232714 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:03:33.232694 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180\": container with ID starting with 491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180 not found: ID does not exist" containerID="491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180" Oct 29 22:03:33.232749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.232723 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180"} err="failed to get container status \"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180\": rpc error: code = NotFound desc = could not find container \"491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180\": container with ID starting with 491ec7a8fdce0d219ae53c5e24588b360e99526d41cb27407402fbfac4e1b180 not found: ID does not exist" Oct 29 22:03:33.232749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.232740 4376 scope.go:122] "RemoveContainer" containerID="4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150" Oct 29 22:03:33.241504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.241481 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\" (UniqueName: \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\") pod \"abc230b1-37be-4146-b032-196373c93cb9\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " Oct 29 22:03:33.241619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.241547 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\") pod \"abc230b1-37be-4146-b032-196373c93cb9\" (UID: \"abc230b1-37be-4146-b032-196373c93cb9\") " Oct 29 22:03:33.241914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.241876 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume" pod "abc230b1-37be-4146-b032-196373c93cb9" (UID: "abc230b1-37be-4146-b032-196373c93cb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:33.242299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.242273 4376 scope.go:122] "RemoveContainer" containerID="4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150" Oct 29 22:03:33.242403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.242299 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc230b1-37be-4146-b032-196373c93cb9-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:33.242403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.242323 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7xv\" (UniqueName: \"kubernetes.io/projected/f5778348-6329-49f9-b2a5-6c9465a2c7e1-kube-api-access-zs7xv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:33.242594 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:03:33.242575 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150\": container with ID starting with 4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150 not found: ID does not exist" containerID="4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150" Oct 29 22:03:33.242630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.242601 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150"} err="failed to get container status \"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150\": rpc error: code = NotFound desc = could not find container \"4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150\": container with ID starting with 4ff04bdaaef71df7573678430e1e3ecbf74090f17080a59b150833eea6ea9150 not found: ID does not exist" Oct 29 22:03:33.243647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.243626 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f" pod "abc230b1-37be-4146-b032-196373c93cb9" (UID: "abc230b1-37be-4146-b032-196373c93cb9"). InnerVolumeSpecName "kube-api-access-x699f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:33.283236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.283205 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:03:33.287266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.287242 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-67bf96f7c8-pjn55"] Oct 29 22:03:33.343698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.343669 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x699f\" (UniqueName: \"kubernetes.io/projected/abc230b1-37be-4146-b032-196373c93cb9-kube-api-access-x699f\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:33.574255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.574224 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:03:33.580857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:33.580830 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-55cfdcc859-jvfhs"] Oct 29 22:03:35.203110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:35.203080 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:03:35.203639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:35.203291 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" containerID="cri-o://730c58e19f27891c2eb9d797f983d16619f2d2e1049b0f0d6c5ef3466a1f08c8" gracePeriod=30 Oct 29 22:03:35.275714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:35.275680 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc230b1-37be-4146-b032-196373c93cb9" path="/var/lib/kubelet/pods/abc230b1-37be-4146-b032-196373c93cb9/volumes" Oct 29 22:03:35.276169 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:35.276153 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" path="/var/lib/kubelet/pods/f5778348-6329-49f9-b2a5-6c9465a2c7e1/volumes" Oct 29 22:03:36.696444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.696413 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:03:36.697191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697172 4376 cpu_manager.go:414] "containerMap: removing container" podUID="abc230b1-37be-4146-b032-196373c93cb9" containerName="payload-pre-processing" Oct 29 22:03:36.697269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697190 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc230b1-37be-4146-b032-196373c93cb9" containerName="payload-pre-processing" Oct 29 22:03:36.697269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697205 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" containerName="payload-processing" Oct 29 22:03:36.697269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697209 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" containerName="payload-processing" Oct 29 22:03:36.697269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697231 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cb13f771-8a74-4195-871a-9a0a4fb1b454" containerName="maas-api" Oct 29 22:03:36.697269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697236 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb13f771-8a74-4195-871a-9a0a4fb1b454" containerName="maas-api" Oct 29 22:03:36.697520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697351 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc230b1-37be-4146-b032-196373c93cb9" containerName="payload-pre-processing" Oct 29 22:03:36.697520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697360 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5778348-6329-49f9-b2a5-6c9465a2c7e1" containerName="payload-processing" Oct 29 22:03:36.697520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.697370 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb13f771-8a74-4195-871a-9a0a4fb1b454" containerName="maas-api" Oct 29 22:03:36.702758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.702734 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.706604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.706490 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.706604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.706567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnjl\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.706770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.706739 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.706825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.706779 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.706886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.706839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.707030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.707009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.707192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.707168 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.707308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.707215 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.707308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.707270 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.725235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.725209 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:03:36.732539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.732514 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fe45b-openshift-default-dockercfg-kkxxf\"" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845560 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845604 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.845796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnjl\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.846048 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.846112 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.846203 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.857926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.846859 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.878923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.860222 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.878923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.870404 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.878923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.870979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.878923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.871230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.892325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.882008 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.892325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.891441 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.941653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.941561 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnjl\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:36.964095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:36.963974 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:37.029876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.029528 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:37.605042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.603558 4376 patch_prober.go:29] interesting pod/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:03:37.605042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.603731 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:03:37.641566 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:03:37.641524 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5f61f2_62d1_47fa_b990_234e45df1f83.slice/crio-d8584dbaab8b47d37ac134ab05ffc5d566b606e0009f25f510a665ef542fba00 WatchSource:0}: Error finding container d8584dbaab8b47d37ac134ab05ffc5d566b606e0009f25f510a665ef542fba00: Status 404 returned error can't find the container with id d8584dbaab8b47d37ac134ab05ffc5d566b606e0009f25f510a665ef542fba00 Oct 29 22:03:37.641756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.641734 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:03:37.644263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.644231 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:37.644334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.644298 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:37.644334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:37.644328 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:38.350470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:38.350434 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerStarted","Data":"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c"} Oct 29 22:03:38.350892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:38.350495 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerStarted","Data":"d8584dbaab8b47d37ac134ab05ffc5d566b606e0009f25f510a665ef542fba00"} Oct 29 22:03:38.449619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:38.448693 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podStartSLOduration=2.4486699659999998 podStartE2EDuration="2.448669966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:38.405769279 +0000 UTC m=+1277.254246124" watchObservedRunningTime="2026-10-29 22:03:38.448669966 +0000 UTC m=+1277.297146812" Oct 29 22:03:39.030688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:39.030649 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:39.037979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:39.037947 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:39.357312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:39.357230 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:39.358054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:39.358032 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:03:40.251344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.251319 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:03:40.255761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.255742 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.276126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.276100 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:03:40.285286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.285260 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ad8e69-serving-cert\"" Oct 29 22:03:40.355643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.355609 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.355811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.355772 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsczn\" (UniqueName: \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.355881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.355812 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.356024 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.356000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.367343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.367315 4376 generic.go:356] "Generic (PLEG): container finished" podID="bf40a427-273f-4d62-96fc-e866a89e9726" containerID="730c58e19f27891c2eb9d797f983d16619f2d2e1049b0f0d6c5ef3466a1f08c8" exitCode=0 Oct 29 22:03:40.367759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.367342 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerDied","Data":"730c58e19f27891c2eb9d797f983d16619f2d2e1049b0f0d6c5ef3466a1f08c8"} Oct 29 22:03:40.367759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.367414 4376 scope.go:122] "RemoveContainer" containerID="da5113c9ecbf316d808f3a989b1427271ebf95f6de3e2c43019727f71a97d280" Oct 29 22:03:40.457789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.457756 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.457971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.457850 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsczn\" (UniqueName: \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.457971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.457881 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.458106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.457985 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.459103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.459077 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.460689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.460669 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.462043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.462021 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.484343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.484215 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsczn\" (UniqueName: \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\") pod \"maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.508529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.508508 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:40.607716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.607682 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:40.662151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662121 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662183 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662266 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662336 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662389 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662499 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662649 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662743 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662779 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\") pod \"bf40a427-273f-4d62-96fc-e866a89e9726\" (UID: \"bf40a427-273f-4d62-96fc-e866a89e9726\") " Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.662834 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.663845 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:40.662929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.663881 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:40.664346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.664005 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:40.664346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.664324 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:40.664552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.664477 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:40.665566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.665542 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "kube-api-access-wkvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:40.665831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.665792 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:03:40.666568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.666538 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:40.666880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.666862 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy" pod "bf40a427-273f-4d62-96fc-e866a89e9726" (UID: "bf40a427-273f-4d62-96fc-e866a89e9726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:40.955342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:40.938103 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.039797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.039770 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf40a427-273f-4d62-96fc-e866a89e9726-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040026 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040060 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf40a427-273f-4d62-96fc-e866a89e9726-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040075 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvm7\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-kube-api-access-wkvm7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040087 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040101 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf40a427-273f-4d62-96fc-e866a89e9726-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.040144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.040112 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf40a427-273f-4d62-96fc-e866a89e9726-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:41.191529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.191499 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:03:41.377106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.377070 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" event={"ID":"c321d293-6700-43ac-be55-6b1f5d067ca7","Type":"ContainerStarted","Data":"6397c8793d1cba4c82e4a4d22f2bfb460983b49e4db5e3cab76a21eac188b3df"} Oct 29 22:03:41.378391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.378371 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" Oct 29 22:03:41.378535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.378391 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f" event={"ID":"bf40a427-273f-4d62-96fc-e866a89e9726","Type":"ContainerDied","Data":"2c5dba2dcf9c4246517c5fb242562867a188c01a525f812628b2d34d1bdd80dd"} Oct 29 22:03:41.378535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.378450 4376 scope.go:122] "RemoveContainer" containerID="730c58e19f27891c2eb9d797f983d16619f2d2e1049b0f0d6c5ef3466a1f08c8" Oct 29 22:03:41.496993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.496859 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:03:41.504368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:41.504337 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-579f09-openshift-default-db9b444c5-w2d7f"] Oct 29 22:03:42.415927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:42.414508 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" event={"ID":"c321d293-6700-43ac-be55-6b1f5d067ca7","Type":"ContainerStarted","Data":"143e631ebe8287404226face3de67a7edfe953020d4d99ebe10c00652c2848d3"} Oct 29 22:03:42.415927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:42.415876 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:42.462987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:42.462898 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" podStartSLOduration=2.04171158 podStartE2EDuration="2.462884579s" totalImagesPullingTime="421.172999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:42.461244187 +0000 UTC m=+1281.309721031" watchObservedRunningTime="2026-10-29 22:03:42.462884579 +0000 UTC m=+1281.311361426" Oct 29 22:03:43.274554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:43.274516 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" path="/var/lib/kubelet/pods/bf40a427-273f-4d62-96fc-e866a89e9726/volumes" Oct 29 22:03:44.392932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.391685 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:03:44.392932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.392814 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:03:44.392932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.392834 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:03:44.418937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.410296 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:03:44.418937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.410339 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:03:44.435926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.426258 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.470926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.455511 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fe45b\"" Oct 29 22:03:44.490941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.481263 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fe45b-dockercfg-2mcsc\"" Oct 29 22:03:44.653512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.653411 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpvl\" (UniqueName: \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.653928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.653888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.758962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.756398 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.758962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.756737 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpvl\" (UniqueName: \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.758962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.758091 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.764311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.764285 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:03:44.788861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.788809 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpvl\" (UniqueName: \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\") pod \"payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:44.940718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:44.940634 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:45.424286 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:03:45.424255 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30badca1_ed66_4178_aa10_1e6246f2d660.slice/crio-ddcc499974a997880caa942a58941c4171341de439e329260354e62ece0b3fcb WatchSource:0}: Error finding container ddcc499974a997880caa942a58941c4171341de439e329260354e62ece0b3fcb: Status 404 returned error can't find the container with id ddcc499974a997880caa942a58941c4171341de439e329260354e62ece0b3fcb Oct 29 22:03:45.426266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:45.426241 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:03:45.516752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:45.516718 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" event={"ID":"30badca1-ed66-4178-aa10-1e6246f2d660","Type":"ContainerStarted","Data":"ddcc499974a997880caa942a58941c4171341de439e329260354e62ece0b3fcb"} Oct 29 22:03:46.480647 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:03:46.480606 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:03:46.524551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:46.524520 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" event={"ID":"30badca1-ed66-4178-aa10-1e6246f2d660","Type":"ContainerStarted","Data":"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded"} Oct 29 22:03:46.524713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:46.524609 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:46.588278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:46.588212 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" podStartSLOduration=2.588198876 podStartE2EDuration="2.588198876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:46.584624648 +0000 UTC m=+1285.433101494" watchObservedRunningTime="2026-10-29 22:03:46.588198876 +0000 UTC m=+1285.436675720" Oct 29 22:03:49.513300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:49.513269 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:03:55.478605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.477439 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:44222->10.134.0.102:15021: read: connection reset by peer" start-of-body= Oct 29 22:03:55.478605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.477510 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:44222->10.134.0.102:15021: read: connection reset by peer" Oct 29 22:03:55.674749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.674661 4376 generic.go:356] "Generic (PLEG): container finished" podID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerID="c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984" exitCode=0 Oct 29 22:03:55.674749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.674702 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerDied","Data":"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984"} Oct 29 22:03:55.675203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.675182 4376 scope.go:122] "RemoveContainer" containerID="c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984" Oct 29 22:03:55.676969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.676936 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:55.677042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.677008 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:55.677076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:55.677042 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:03:56.684669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:56.684629 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerStarted","Data":"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160"} Oct 29 22:03:57.555928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:57.555072 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:03:57.632437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:57.632401 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:03:57.632757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:57.632727 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:57.632953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:57.632786 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 29 22:03:58.632774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:58.632739 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:58.633267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:58.632807 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 29 22:03:59.674744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:59.674705 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:59.675325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:59.675290 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 29 22:03:59.906927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:59.900176 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:03:59.942972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:59.935483 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:03:59.942972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:03:59.938952 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf40a427-273f-4d62-96fc-e866a89e9726" containerName="istio-proxy" Oct 29 22:04:00.091224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.091190 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:04:00.091510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.091486 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:04:00.099590 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.098028 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.138126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.137982 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d9f8c0-dockercfg-vrh6s\"" Oct 29 22:04:00.138126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.138092 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d9f8c0\"" Oct 29 22:04:00.148936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.147522 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:04:00.148936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.147650 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.270946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.267119 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.270946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.270206 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vcd\" (UniqueName: \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.270946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.270237 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ngx\" (UniqueName: \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.270946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.270260 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.384067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.384029 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vcd\" (UniqueName: \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.384401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.384373 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ngx\" (UniqueName: \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.384512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.384440 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.384645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.384619 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.385700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.385676 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.386580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.386531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.410893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.410842 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vcd\" (UniqueName: \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\") pod \"payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.412981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.412960 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ngx\" (UniqueName: \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\") pod \"payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.448258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.443219 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:00.513933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.512286 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:00.717715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.700746 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:04:00.717715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:00.701229 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 29 22:04:01.126762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.126659 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:04:01.188382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.188347 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:04:01.193322 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:04:01.193295 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950d2aef_54a7_444f_8075_3b0a0309c7df.slice/crio-7e8664ee2eafe21d324377686d2e6ced79272aac8e6b8af5afc66d857a64cf3d WatchSource:0}: Error finding container 7e8664ee2eafe21d324377686d2e6ced79272aac8e6b8af5afc66d857a64cf3d: Status 404 returned error can't find the container with id 7e8664ee2eafe21d324377686d2e6ced79272aac8e6b8af5afc66d857a64cf3d Oct 29 22:04:01.632742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.632642 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:04:01.632742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.632717 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 29 22:04:01.812764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.812725 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" event={"ID":"950d2aef-54a7-444f-8075-3b0a0309c7df","Type":"ContainerStarted","Data":"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9"} Oct 29 22:04:01.813196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.812794 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:01.813196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.812814 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" event={"ID":"950d2aef-54a7-444f-8075-3b0a0309c7df","Type":"ContainerStarted","Data":"7e8664ee2eafe21d324377686d2e6ced79272aac8e6b8af5afc66d857a64cf3d"} Oct 29 22:04:01.814683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.814657 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" event={"ID":"fa1b280d-39a9-406d-afa4-7bbbf7896a28","Type":"ContainerStarted","Data":"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf"} Oct 29 22:04:01.814845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.814696 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" event={"ID":"fa1b280d-39a9-406d-afa4-7bbbf7896a28","Type":"ContainerStarted","Data":"24f79fb3b4adb18e50893d40d7a27385b8191299c48e934c8677e334106549ba"} Oct 29 22:04:01.814845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.814736 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:01.874575 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.874499 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" podStartSLOduration=2.874476175 podStartE2EDuration="2.874476175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:01.8645867 +0000 UTC m=+1300.713063546" watchObservedRunningTime="2026-10-29 22:04:01.874476175 +0000 UTC m=+1300.722953042" Oct 29 22:04:01.961341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:01.948300 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" podStartSLOduration=2.948284466 podStartE2EDuration="2.948284466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:01.946269011 +0000 UTC m=+1300.794745861" watchObservedRunningTime="2026-10-29 22:04:01.948284466 +0000 UTC m=+1300.796761324" Oct 29 22:04:02.692694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:02.692657 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:04:02.693159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:02.693134 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:04:02.694573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:02.694550 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:04:09.909215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.909187 4376 generic.go:356] "Generic (PLEG): container finished" podID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerID="e7ca71e58a9d5c412bc781f96652d426fecea53fe809879673e8ee1005ec71be" exitCode=0 Oct 29 22:04:09.909545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.909258 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerDied","Data":"e7ca71e58a9d5c412bc781f96652d426fecea53fe809879673e8ee1005ec71be"} Oct 29 22:04:09.909639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.909625 4376 scope.go:122] "RemoveContainer" containerID="e7ca71e58a9d5c412bc781f96652d426fecea53fe809879673e8ee1005ec71be" Oct 29 22:04:09.911594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.911562 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:09.911713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.911644 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:09.911713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:09.911690 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:10.918793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:10.918760 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerStarted","Data":"db53580a52d570447057f23f41fde29f7f7e95cb03c035f434f9960e3ee2c309"} Oct 29 22:04:11.306230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:11.306197 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:04:11.307625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:11.307599 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:11.307740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:11.307645 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:12.316682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:12.310222 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:12.316682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:12.310332 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:12.830967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:12.830202 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:04:12.845990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:12.831953 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:04:13.309925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:13.308878 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:13.309925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:13.308970 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:14.382927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:14.382146 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:14.382927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:14.382219 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:14.627711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:14.621054 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:04:15.306196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:15.306160 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:15.306368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:15.306213 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:16.306645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:16.306611 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:04:16.307085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:16.306666 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 29 22:04:16.435466 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:16.435413 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:04:17.311776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:17.311745 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:04:17.313000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:17.312982 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:04:18.977839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:18.977799 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 22:04:18.978535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:18.978504 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" containerID="cri-o://54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f" gracePeriod=30 Oct 29 22:04:18.991522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:18.991494 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 22:04:18.991726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:18.991703 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" containerID="cri-o://fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f" gracePeriod=30 Oct 29 22:04:20.838953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:20.838844 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:04:20.839425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:20.839109 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" podUID="a610ae81-2434-4735-b850-2818a82f478a" containerName="maas-api" containerID="cri-o://707eaba34a3a0b8b68f0747676fb56c8c6bdcb370e15c6602bf1a0122f8ab89d" gracePeriod=30 Oct 29 22:04:21.689957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:21.680968 4376 generic.go:356] "Generic (PLEG): container finished" podID="a610ae81-2434-4735-b850-2818a82f478a" containerID="707eaba34a3a0b8b68f0747676fb56c8c6bdcb370e15c6602bf1a0122f8ab89d" exitCode=0 Oct 29 22:04:21.689957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:21.688241 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" event={"ID":"a610ae81-2434-4735-b850-2818a82f478a","Type":"ContainerDied","Data":"707eaba34a3a0b8b68f0747676fb56c8c6bdcb370e15c6602bf1a0122f8ab89d"} Oct 29 22:04:22.065023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.064994 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:04:22.153108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.153073 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\") pod \"a610ae81-2434-4735-b850-2818a82f478a\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " Oct 29 22:04:22.153299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.153131 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\" (UniqueName: \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\") pod \"a610ae81-2434-4735-b850-2818a82f478a\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " Oct 29 22:04:22.153612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.153590 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\") pod \"a610ae81-2434-4735-b850-2818a82f478a\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " Oct 29 22:04:22.153696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.153657 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\") pod \"a610ae81-2434-4735-b850-2818a82f478a\" (UID: \"a610ae81-2434-4735-b850-2818a82f478a\") " Oct 29 22:04:22.154652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.154143 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca" pod "a610ae81-2434-4735-b850-2818a82f478a" (UID: "a610ae81-2434-4735-b850-2818a82f478a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:22.154652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.154622 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a610ae81-2434-4735-b850-2818a82f478a-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:22.157371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.157329 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk" pod "a610ae81-2434-4735-b850-2818a82f478a" (UID: "a610ae81-2434-4735-b850-2818a82f478a"). InnerVolumeSpecName "kube-api-access-jckhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:22.158734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.158707 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls" pod "a610ae81-2434-4735-b850-2818a82f478a" (UID: "a610ae81-2434-4735-b850-2818a82f478a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:22.158854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.158779 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs" pod "a610ae81-2434-4735-b850-2818a82f478a" (UID: "a610ae81-2434-4735-b850-2818a82f478a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:22.256497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.256464 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:22.256497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.256495 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a610ae81-2434-4735-b850-2818a82f478a-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:22.256497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.256504 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckhk\" (UniqueName: \"kubernetes.io/projected/a610ae81-2434-4735-b850-2818a82f478a-kube-api-access-jckhk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:22.696870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.696837 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" Oct 29 22:04:22.696870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.696863 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl" event={"ID":"a610ae81-2434-4735-b850-2818a82f478a","Type":"ContainerDied","Data":"cec427f4ae60641a376f25cb6f364a74e62008a76f7a5454da59e7468160dbbc"} Oct 29 22:04:22.697167 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.696948 4376 scope.go:122] "RemoveContainer" containerID="707eaba34a3a0b8b68f0747676fb56c8c6bdcb370e15c6602bf1a0122f8ab89d" Oct 29 22:04:22.758061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.758020 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:04:22.771277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:22.771242 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8867e7d-74777f4cf9-957xl"] Oct 29 22:04:23.274051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:23.274019 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a610ae81-2434-4735-b850-2818a82f478a" path="/var/lib/kubelet/pods/a610ae81-2434-4735-b850-2818a82f478a/volumes" Oct 29 22:04:24.384517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.384471 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 22:04:24.438032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.437992 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438084 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438143 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438179 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438365 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438425 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438466 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438568 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.438646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438608 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.439000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.438703 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\") pod \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\" (UID: \"5bc7edf8-edf6-45fc-aea4-4228f96c231b\") " Oct 29 22:04:24.440253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.439394 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:24.440253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.439581 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.440253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.439796 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.440253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.439808 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.440253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.439817 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.440743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.440721 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.441638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.441602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.442512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.442472 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.444304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.444271 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.444766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.444733 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:24.450836 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.450812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp" pod "5bc7edf8-edf6-45fc-aea4-4228f96c231b" (UID: "5bc7edf8-edf6-45fc-aea4-4228f96c231b"). InnerVolumeSpecName "kube-api-access-5k9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.468166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.468099 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 22:04:24.541184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541152 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541215 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541253 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541345 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541375 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.541396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541394 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541578 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.541730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541702 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.541730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541650 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:24.541837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541742 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541779 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541843 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.541968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.541931 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\") pod \"d006d9b8-4c3d-474d-8667-9779d742c8ff\" (UID: \"d006d9b8-4c3d-474d-8667-9779d742c8ff\") " Oct 29 22:04:24.542079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542034 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.542433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542412 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542430 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542440 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542449 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542457 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542465 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d006d9b8-4c3d-474d-8667-9779d742c8ff-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542474 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542485 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc7edf8-edf6-45fc-aea4-4228f96c231b-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542497 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542509 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc7edf8-edf6-45fc-aea4-4228f96c231b-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.542613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.542519 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k9jp\" (UniqueName: \"kubernetes.io/projected/5bc7edf8-edf6-45fc-aea4-4228f96c231b-kube-api-access-5k9jp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.544384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.544357 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.545198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.545174 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:24.545303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.545269 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "kube-api-access-ln45d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.546623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.546599 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token" pod "d006d9b8-4c3d-474d-8667-9779d742c8ff" (UID: "d006d9b8-4c3d-474d-8667-9779d742c8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.652204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.650602 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln45d\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-kube-api-access-ln45d\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.652204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.650652 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.652204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.650669 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.652204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.650682 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d006d9b8-4c3d-474d-8667-9779d742c8ff-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.712370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.712339 4376 generic.go:356] "Generic (PLEG): container finished" podID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerID="fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f" exitCode=0 Oct 29 22:04:24.712577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.712384 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" Oct 29 22:04:24.712577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.712417 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" event={"ID":"d006d9b8-4c3d-474d-8667-9779d742c8ff","Type":"ContainerDied","Data":"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f"} Oct 29 22:04:24.712577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.712490 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5" event={"ID":"d006d9b8-4c3d-474d-8667-9779d742c8ff","Type":"ContainerDied","Data":"e539f858e916d705b2f68f052151c8859f75b3c60b56aa03fe03594455282977"} Oct 29 22:04:24.712577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.712522 4376 scope.go:122] "RemoveContainer" containerID="fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f" Oct 29 22:04:24.714106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.714083 4376 generic.go:356] "Generic (PLEG): container finished" podID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerID="54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f" exitCode=0 Oct 29 22:04:24.714219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.714142 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" Oct 29 22:04:24.714219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.714164 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" event={"ID":"5bc7edf8-edf6-45fc-aea4-4228f96c231b","Type":"ContainerDied","Data":"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f"} Oct 29 22:04:24.714219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.714204 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf" event={"ID":"5bc7edf8-edf6-45fc-aea4-4228f96c231b","Type":"ContainerDied","Data":"d8785afc25939685d4f8cbec6eb38bad26a5c1bac83fb74256076b619cb1f2bb"} Oct 29 22:04:24.723453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.723432 4376 scope.go:122] "RemoveContainer" containerID="fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f" Oct 29 22:04:24.723736 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:24.723715 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f\": container with ID starting with fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f not found: ID does not exist" containerID="fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f" Oct 29 22:04:24.723839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.723743 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f"} err="failed to get container status \"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f\": rpc error: code = NotFound desc = could not find container \"fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f\": container with ID starting with fcb0ce792908b6be963126991dfcab41e8b4b14406905d1209c4641e2d00894f not found: ID does not exist" Oct 29 22:04:24.723839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.723759 4376 scope.go:122] "RemoveContainer" containerID="54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f" Oct 29 22:04:24.733570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.733553 4376 scope.go:122] "RemoveContainer" containerID="54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f" Oct 29 22:04:24.733817 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:24.733800 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f\": container with ID starting with 54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f not found: ID does not exist" containerID="54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f" Oct 29 22:04:24.733870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.733825 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f"} err="failed to get container status \"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f\": rpc error: code = NotFound desc = could not find container \"54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f\": container with ID starting with 54d8965387f3876bdf703b09d6aaad38495d38952ca33d0070a733329fbd6a9f not found: ID does not exist" Oct 29 22:04:24.774518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.774478 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 22:04:24.781448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.781427 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-5fnvf"] Oct 29 22:04:24.820208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.820181 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 22:04:24.825227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:24.825208 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-lvkx5"] Oct 29 22:04:25.343875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:25.343534 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" path="/var/lib/kubelet/pods/5bc7edf8-edf6-45fc-aea4-4228f96c231b/volumes" Oct 29 22:04:25.345191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:25.345156 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" path="/var/lib/kubelet/pods/d006d9b8-4c3d-474d-8667-9779d742c8ff/volumes" Oct 29 22:04:38.132211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.132176 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:04:38.132786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.132423 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" podUID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" containerName="maas-api" containerID="cri-o://7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc" gracePeriod=30 Oct 29 22:04:38.667081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.666691 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:04:38.784629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.784560 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\") pod \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " Oct 29 22:04:38.784853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.784765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\" (UniqueName: \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\") pod \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " Oct 29 22:04:38.784943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.784867 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\") pod \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " Oct 29 22:04:38.785011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.784937 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\") pod \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\" (UID: \"7665687b-bde5-4c7a-a654-5ed89dd89ca2\") " Oct 29 22:04:38.786882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.786842 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca" pod "7665687b-bde5-4c7a-a654-5ed89dd89ca2" (UID: "7665687b-bde5-4c7a-a654-5ed89dd89ca2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:38.797818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.797744 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs" pod "7665687b-bde5-4c7a-a654-5ed89dd89ca2" (UID: "7665687b-bde5-4c7a-a654-5ed89dd89ca2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:38.798160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.798132 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls" pod "7665687b-bde5-4c7a-a654-5ed89dd89ca2" (UID: "7665687b-bde5-4c7a-a654-5ed89dd89ca2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:38.798861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.798838 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct" pod "7665687b-bde5-4c7a-a654-5ed89dd89ca2" (UID: "7665687b-bde5-4c7a-a654-5ed89dd89ca2"). InnerVolumeSpecName "kube-api-access-pkqct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:38.815415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.807281 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:38.815415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.807315 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7665687b-bde5-4c7a-a654-5ed89dd89ca2-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:38.815415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.807330 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7665687b-bde5-4c7a-a654-5ed89dd89ca2-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:38.815415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.807344 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqct\" (UniqueName: \"kubernetes.io/projected/7665687b-bde5-4c7a-a654-5ed89dd89ca2-kube-api-access-pkqct\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:38.983620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.983583 4376 generic.go:356] "Generic (PLEG): container finished" podID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" containerID="7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc" exitCode=0 Oct 29 22:04:38.983792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.983634 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" Oct 29 22:04:38.983792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.983658 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" event={"ID":"7665687b-bde5-4c7a-a654-5ed89dd89ca2","Type":"ContainerDied","Data":"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc"} Oct 29 22:04:38.983792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.983712 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk" event={"ID":"7665687b-bde5-4c7a-a654-5ed89dd89ca2","Type":"ContainerDied","Data":"ef9ea42e33caeaba62856b5a1684c4195c48a117d28d1b0f6aff263774208c1c"} Oct 29 22:04:38.983792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.983771 4376 scope.go:122] "RemoveContainer" containerID="7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc" Oct 29 22:04:38.995129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.995109 4376 scope.go:122] "RemoveContainer" containerID="7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc" Oct 29 22:04:38.995507 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:38.995449 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc\": container with ID starting with 7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc not found: ID does not exist" containerID="7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc" Oct 29 22:04:38.995507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:38.995480 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc"} err="failed to get container status \"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc\": rpc error: code = NotFound desc = could not find container \"7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc\": container with ID starting with 7b7555937ca065e2254ba77314b2add4b07655524f1c640582ae52dd6adb46fc not found: ID does not exist" Oct 29 22:04:39.045770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:39.045741 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:04:39.058329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:39.058264 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fcce6b-786c676ffb-kjntk"] Oct 29 22:04:39.289063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:39.288291 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" path="/var/lib/kubelet/pods/7665687b-bde5-4c7a-a654-5ed89dd89ca2/volumes" Oct 29 22:04:46.493596 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:46.493560 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:04:47.389261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:47.389187 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:04:48.513605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.509970 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:04:48.549928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.537162 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" Oct 29 22:04:48.550176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.550147 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" Oct 29 22:04:48.550334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.550319 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" Oct 29 22:04:48.550444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.550429 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" Oct 29 22:04:48.558941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.550563 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a610ae81-2434-4735-b850-2818a82f478a" containerName="maas-api" Oct 29 22:04:48.559179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.559154 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a610ae81-2434-4735-b850-2818a82f478a" containerName="maas-api" Oct 29 22:04:48.566932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.559320 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" containerName="maas-api" Oct 29 22:04:48.609005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.608963 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" containerName="maas-api" Oct 29 22:04:48.636961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.623258 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7665687b-bde5-4c7a-a654-5ed89dd89ca2" containerName="maas-api" Oct 29 22:04:48.637315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.637288 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc7edf8-edf6-45fc-aea4-4228f96c231b" containerName="istio-proxy" Oct 29 22:04:48.637446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.637436 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d006d9b8-4c3d-474d-8667-9779d742c8ff" containerName="istio-proxy" Oct 29 22:04:48.796110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.738000 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a610ae81-2434-4735-b850-2818a82f478a" containerName="maas-api" Oct 29 22:04:48.796110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.743043 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:04:48.796110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.743170 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.813321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.812699 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad8e69\"" Oct 29 22:04:48.813321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.812806 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad8e69-dockercfg-kkthl\"" Oct 29 22:04:48.848952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.839472 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.848952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.839606 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcz6d\" (UniqueName: \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.942927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.941090 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.942927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.941239 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcz6d\" (UniqueName: \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.942927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.942605 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:48.967793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:48.967763 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcz6d\" (UniqueName: \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:49.094397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.093956 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:04:49.094397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.094248 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" podUID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" containerName="maas-api" containerID="cri-o://c015161bc54f7fe38a97997088796b0c7b05f0c38e5a00a86c645efe9e9f1496" gracePeriod=30 Oct 29 22:04:49.123600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.123573 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:49.192204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.192168 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:04:49.192474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.192406 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" podUID="97489e7f-829c-4041-bf51-ef7ba61692c1" containerName="payload-processing" containerID="cri-o://639114cbfebcfcd1cf45d1126eeac327e0c47eea94ebd267b31c9933298de2e5" gracePeriod=30 Oct 29 22:04:49.214068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.213747 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:04:49.214068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.214005 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" podUID="b8c1b211-4366-492e-af83-5b1fd097c8d9" containerName="payload-pre-processing" containerID="cri-o://3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de" gracePeriod=30 Oct 29 22:04:49.326238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.326196 4376 generic.go:356] "Generic (PLEG): container finished" podID="97489e7f-829c-4041-bf51-ef7ba61692c1" containerID="639114cbfebcfcd1cf45d1126eeac327e0c47eea94ebd267b31c9933298de2e5" exitCode=0 Oct 29 22:04:49.326435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.326340 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" event={"ID":"97489e7f-829c-4041-bf51-ef7ba61692c1","Type":"ContainerDied","Data":"639114cbfebcfcd1cf45d1126eeac327e0c47eea94ebd267b31c9933298de2e5"} Oct 29 22:04:49.329506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.329480 4376 generic.go:356] "Generic (PLEG): container finished" podID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" containerID="c015161bc54f7fe38a97997088796b0c7b05f0c38e5a00a86c645efe9e9f1496" exitCode=0 Oct 29 22:04:49.329666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.329649 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" event={"ID":"6cbbcea9-28a9-4221-a46b-98edd17d27e9","Type":"ContainerDied","Data":"c015161bc54f7fe38a97997088796b0c7b05f0c38e5a00a86c645efe9e9f1496"} Oct 29 22:04:49.663564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.663525 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:04:49.859112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.818938 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:04:49.964001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.932060 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\") pod \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " Oct 29 22:04:49.964001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.932303 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\") pod \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " Oct 29 22:04:49.964001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.932361 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\" (UniqueName: \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\") pod \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " Oct 29 22:04:49.964001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.932498 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\") pod \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\" (UID: \"6cbbcea9-28a9-4221-a46b-98edd17d27e9\") " Oct 29 22:04:49.975958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.975352 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls" pod "6cbbcea9-28a9-4221-a46b-98edd17d27e9" (UID: "6cbbcea9-28a9-4221-a46b-98edd17d27e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:49.981944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.981891 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd" pod "6cbbcea9-28a9-4221-a46b-98edd17d27e9" (UID: "6cbbcea9-28a9-4221-a46b-98edd17d27e9"). InnerVolumeSpecName "kube-api-access-6xhfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:49.982466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:49.982435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca" pod "6cbbcea9-28a9-4221-a46b-98edd17d27e9" (UID: "6cbbcea9-28a9-4221-a46b-98edd17d27e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:50.013656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.010115 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs" pod "6cbbcea9-28a9-4221-a46b-98edd17d27e9" (UID: "6cbbcea9-28a9-4221-a46b-98edd17d27e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:50.110460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.049014 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbbcea9-28a9-4221-a46b-98edd17d27e9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.110460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.049053 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhfd\" (UniqueName: \"kubernetes.io/projected/6cbbcea9-28a9-4221-a46b-98edd17d27e9-kube-api-access-6xhfd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.110460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.049069 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.110460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.049085 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbbcea9-28a9-4221-a46b-98edd17d27e9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.270544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.270298 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:04:50.304737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.304646 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:04:50.373056 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.373017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" event={"ID":"97489e7f-829c-4041-bf51-ef7ba61692c1","Type":"ContainerDied","Data":"b67a427386d541eae80f14fafb9bac95b1615a523988c62dca6251b35f5500ec"} Oct 29 22:04:50.373248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.373091 4376 scope.go:122] "RemoveContainer" containerID="639114cbfebcfcd1cf45d1126eeac327e0c47eea94ebd267b31c9933298de2e5" Oct 29 22:04:50.373248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.373032 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d" Oct 29 22:04:50.375230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.375216 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" Oct 29 22:04:50.375453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.375273 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" event={"ID":"b8c1b211-4366-492e-af83-5b1fd097c8d9","Type":"ContainerDied","Data":"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de"} Oct 29 22:04:50.375624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.375161 4376 generic.go:356] "Generic (PLEG): container finished" podID="b8c1b211-4366-492e-af83-5b1fd097c8d9" containerID="3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de" exitCode=0 Oct 29 22:04:50.375869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.375853 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw" event={"ID":"b8c1b211-4366-492e-af83-5b1fd097c8d9","Type":"ContainerDied","Data":"dbee440b897c128691c713bf44b13e67a191101d537266f4641a95ea179c2601"} Oct 29 22:04:50.377860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.377808 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" event={"ID":"adf52120-be10-4efb-99a4-9d18f6e70dbc","Type":"ContainerStarted","Data":"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae"} Oct 29 22:04:50.377860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.377840 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" event={"ID":"adf52120-be10-4efb-99a4-9d18f6e70dbc","Type":"ContainerStarted","Data":"99a7e9ab24f28dd46a398b96203c31f0faf740ece58a1a8caea8fd084448993d"} Oct 29 22:04:50.380227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.380213 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" event={"ID":"6cbbcea9-28a9-4221-a46b-98edd17d27e9","Type":"ContainerDied","Data":"4da53f38fba80cbf1fa1e2443491ad99d9b5353753e307a886af1d1ebbc44822"} Oct 29 22:04:50.380383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.380364 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs" Oct 29 22:04:50.384992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.384972 4376 scope.go:122] "RemoveContainer" containerID="3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de" Oct 29 22:04:50.403606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.403576 4376 scope.go:122] "RemoveContainer" containerID="3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de" Oct 29 22:04:50.404287 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:50.404176 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de\": container with ID starting with 3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de not found: ID does not exist" containerID="3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de" Oct 29 22:04:50.404287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.404207 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de"} err="failed to get container status \"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de\": rpc error: code = NotFound desc = could not find container \"3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de\": container with ID starting with 3a7b13d77009ef5f3ba32ff6773cd4a75e005b7b23a0b3ceaf4e8c95e8b5a6de not found: ID does not exist" Oct 29 22:04:50.404287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.404225 4376 scope.go:122] "RemoveContainer" containerID="c015161bc54f7fe38a97997088796b0c7b05f0c38e5a00a86c645efe9e9f1496" Oct 29 22:04:50.406147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.406124 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\") pod \"97489e7f-829c-4041-bf51-ef7ba61692c1\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " Oct 29 22:04:50.406226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.406191 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\" (UniqueName: \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\") pod \"97489e7f-829c-4041-bf51-ef7ba61692c1\" (UID: \"97489e7f-829c-4041-bf51-ef7ba61692c1\") " Oct 29 22:04:50.406296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.406267 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\" (UniqueName: \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\") pod \"b8c1b211-4366-492e-af83-5b1fd097c8d9\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " Oct 29 22:04:50.406363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.406338 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\") pod \"b8c1b211-4366-492e-af83-5b1fd097c8d9\" (UID: \"b8c1b211-4366-492e-af83-5b1fd097c8d9\") " Oct 29 22:04:50.407034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.406828 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume" pod "97489e7f-829c-4041-bf51-ef7ba61692c1" (UID: "97489e7f-829c-4041-bf51-ef7ba61692c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:50.407251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.407230 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume" pod "b8c1b211-4366-492e-af83-5b1fd097c8d9" (UID: "b8c1b211-4366-492e-af83-5b1fd097c8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:50.407598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.407581 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97489e7f-829c-4041-bf51-ef7ba61692c1-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.407681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.407602 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c1b211-4366-492e-af83-5b1fd097c8d9-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.409298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.409276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv" pod "97489e7f-829c-4041-bf51-ef7ba61692c1" (UID: "97489e7f-829c-4041-bf51-ef7ba61692c1"). InnerVolumeSpecName "kube-api-access-kfdtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:50.409714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.409682 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w" pod "b8c1b211-4366-492e-af83-5b1fd097c8d9" (UID: "b8c1b211-4366-492e-af83-5b1fd097c8d9"). InnerVolumeSpecName "kube-api-access-ntf8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:50.471204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.471114 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:04:50.474865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.474835 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76c786-77b6ffc4dc-twrzs"] Oct 29 22:04:50.508916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.508874 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdtv\" (UniqueName: \"kubernetes.io/projected/97489e7f-829c-4041-bf51-ef7ba61692c1-kube-api-access-kfdtv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.509082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.508932 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntf8w\" (UniqueName: \"kubernetes.io/projected/b8c1b211-4366-492e-af83-5b1fd097c8d9-kube-api-access-ntf8w\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:50.564291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.564226 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" podStartSLOduration=2.564211818 podStartE2EDuration="2.564211818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:50.557385909 +0000 UTC m=+1349.405862757" watchObservedRunningTime="2026-10-29 22:04:50.564211818 +0000 UTC m=+1349.412688662" Oct 29 22:04:50.827160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.827005 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:04:50.837927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.833165 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-78d8fbcf55-5wsxw"] Oct 29 22:04:50.891171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.891135 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:04:50.963455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:50.963394 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-7cc64495d8-wl29d"] Oct 29 22:04:51.285453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:51.285414 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" path="/var/lib/kubelet/pods/6cbbcea9-28a9-4221-a46b-98edd17d27e9/volumes" Oct 29 22:04:51.285985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:51.285967 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97489e7f-829c-4041-bf51-ef7ba61692c1" path="/var/lib/kubelet/pods/97489e7f-829c-4041-bf51-ef7ba61692c1/volumes" Oct 29 22:04:51.286380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:51.286367 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c1b211-4366-492e-af83-5b1fd097c8d9" path="/var/lib/kubelet/pods/b8c1b211-4366-492e-af83-5b1fd097c8d9/volumes" Oct 29 22:04:51.391343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:51.391314 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:04:52.390000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.389954 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" podUID="30badca1-ed66-4178-aa10-1e6246f2d660" containerName="payload-processing" containerID="cri-o://09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded" gracePeriod=25 Oct 29 22:04:52.795896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.795865 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:04:52.885687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.885655 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\" (UniqueName: \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\") pod \"30badca1-ed66-4178-aa10-1e6246f2d660\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " Oct 29 22:04:52.886069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.886045 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\") pod \"30badca1-ed66-4178-aa10-1e6246f2d660\" (UID: \"30badca1-ed66-4178-aa10-1e6246f2d660\") " Oct 29 22:04:52.886582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.886555 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume" pod "30badca1-ed66-4178-aa10-1e6246f2d660" (UID: "30badca1-ed66-4178-aa10-1e6246f2d660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:52.886868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.886842 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30badca1-ed66-4178-aa10-1e6246f2d660-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.888188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.888164 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl" pod "30badca1-ed66-4178-aa10-1e6246f2d660" (UID: "30badca1-ed66-4178-aa10-1e6246f2d660"). InnerVolumeSpecName "kube-api-access-krpvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:52.988774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:52.988742 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krpvl\" (UniqueName: \"kubernetes.io/projected/30badca1-ed66-4178-aa10-1e6246f2d660-kube-api-access-krpvl\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:53.446684 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.446646 4376 generic.go:356] "Generic (PLEG): container finished" podID="30badca1-ed66-4178-aa10-1e6246f2d660" containerID="09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded" exitCode=0 Oct 29 22:04:53.447336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.447307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" event={"ID":"30badca1-ed66-4178-aa10-1e6246f2d660","Type":"ContainerDied","Data":"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded"} Oct 29 22:04:53.447496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.447481 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" event={"ID":"30badca1-ed66-4178-aa10-1e6246f2d660","Type":"ContainerDied","Data":"ddcc499974a997880caa942a58941c4171341de439e329260354e62ece0b3fcb"} Oct 29 22:04:53.447616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.447601 4376 scope.go:122] "RemoveContainer" containerID="09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded" Oct 29 22:04:53.447860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.447841 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq" Oct 29 22:04:53.481330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.481107 4376 scope.go:122] "RemoveContainer" containerID="09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded" Oct 29 22:04:53.482698 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:53.482603 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded\": container with ID starting with 09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded not found: ID does not exist" containerID="09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded" Oct 29 22:04:53.484076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.484033 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded"} err="failed to get container status \"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded\": rpc error: code = NotFound desc = could not find container \"09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded\": container with ID starting with 09026066f446cb27cd81aec595e436cd35907fd6351d52edfa5bbeb8c4f71ded not found: ID does not exist" Oct 29 22:04:53.680976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.672965 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:04:53.709011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.708178 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-c8b8c6c7d-h69pq"] Oct 29 22:04:53.845929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.845626 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846841 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" containerName="maas-api" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846870 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" containerName="maas-api" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846940 4376 cpu_manager.go:414] "containerMap: removing container" podUID="30badca1-ed66-4178-aa10-1e6246f2d660" containerName="payload-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846949 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30badca1-ed66-4178-aa10-1e6246f2d660" containerName="payload-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846981 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b8c1b211-4366-492e-af83-5b1fd097c8d9" containerName="payload-pre-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.846989 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c1b211-4366-492e-af83-5b1fd097c8d9" containerName="payload-pre-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847010 4376 cpu_manager.go:414] "containerMap: removing container" podUID="97489e7f-829c-4041-bf51-ef7ba61692c1" containerName="payload-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847018 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97489e7f-829c-4041-bf51-ef7ba61692c1" containerName="payload-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847171 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c1b211-4366-492e-af83-5b1fd097c8d9" containerName="payload-pre-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847186 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="97489e7f-829c-4041-bf51-ef7ba61692c1" containerName="payload-processing" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847199 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbbcea9-28a9-4221-a46b-98edd17d27e9" containerName="maas-api" Oct 29 22:04:53.861574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:53.847215 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="30badca1-ed66-4178-aa10-1e6246f2d660" containerName="payload-processing" Oct 29 22:04:54.142130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.142088 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:04:54.142509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.142485 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.166813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.166770 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-442dc2-dockercfg-qfb5r\"" Oct 29 22:04:54.191173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.185042 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:04:54.191438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.191310 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-442dc2\"" Oct 29 22:04:54.198536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.198506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:04:54.198687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.198630 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.270826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.270792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6v5x\" (UniqueName: \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.271092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.270960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.271145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.271134 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.271189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.271174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7jf\" (UniqueName: \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.373002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.372964 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.373207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.373036 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7jf\" (UniqueName: \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.373207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.373124 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6v5x\" (UniqueName: \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.373336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.373229 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.373798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.373773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.374022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.374002 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.399448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.399380 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6v5x\" (UniqueName: \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.400522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.400495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7jf\" (UniqueName: \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\") pod \"payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.499198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.497939 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:54.516233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.516204 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:54.869139 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:04:54.869100 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d83b5c_eace_4a54_b4b1_39cb143c35ad.slice/crio-26aba0b0fda782b7c2ef25f75c253cab317add9877b025e6a83d2d6c0bb3fa66 WatchSource:0}: Error finding container 26aba0b0fda782b7c2ef25f75c253cab317add9877b025e6a83d2d6c0bb3fa66: Status 404 returned error can't find the container with id 26aba0b0fda782b7c2ef25f75c253cab317add9877b025e6a83d2d6c0bb3fa66 Oct 29 22:04:54.870072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.870009 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:04:54.903735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:54.903708 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:04:54.906685 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:04:54.906659 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994c957_9487_4b4c_a890_1400fc54b265.slice/crio-89e70e5eac5273bbf450ebcd28455c8bef507608438ac13bca9374ec7378ba6d WatchSource:0}: Error finding container 89e70e5eac5273bbf450ebcd28455c8bef507608438ac13bca9374ec7378ba6d: Status 404 returned error can't find the container with id 89e70e5eac5273bbf450ebcd28455c8bef507608438ac13bca9374ec7378ba6d Oct 29 22:04:55.276195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.276158 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30badca1-ed66-4178-aa10-1e6246f2d660" path="/var/lib/kubelet/pods/30badca1-ed66-4178-aa10-1e6246f2d660/volumes" Oct 29 22:04:55.398112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.398067 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:04:55.398315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.398131 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:04:55.467868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.467829 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" event={"ID":"1994c957-9487-4b4c-a890-1400fc54b265","Type":"ContainerStarted","Data":"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14"} Oct 29 22:04:55.468078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.467895 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" event={"ID":"1994c957-9487-4b4c-a890-1400fc54b265","Type":"ContainerStarted","Data":"89e70e5eac5273bbf450ebcd28455c8bef507608438ac13bca9374ec7378ba6d"} Oct 29 22:04:55.468078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.467963 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:04:55.469711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.469683 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" event={"ID":"89d83b5c-eace-4a54-b4b1-39cb143c35ad","Type":"ContainerStarted","Data":"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a"} Oct 29 22:04:55.469826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.469734 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" event={"ID":"89d83b5c-eace-4a54-b4b1-39cb143c35ad","Type":"ContainerStarted","Data":"26aba0b0fda782b7c2ef25f75c253cab317add9877b025e6a83d2d6c0bb3fa66"} Oct 29 22:04:55.469826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.469761 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:04:55.561218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:55.561123 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" podStartSLOduration=2.561107494 podStartE2EDuration="2.561107494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:55.515878788 +0000 UTC m=+1354.364355672" watchObservedRunningTime="2026-10-29 22:04:55.561107494 +0000 UTC m=+1354.409584339" Oct 29 22:04:56.114842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.114787 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" podStartSLOduration=3.114772275 podStartE2EDuration="3.114772275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:55.564699305 +0000 UTC m=+1354.413176148" watchObservedRunningTime="2026-10-29 22:04:56.114772275 +0000 UTC m=+1354.963249119" Oct 29 22:04:56.117098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.117076 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:04:56.117289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.117267 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" podUID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" containerName="payload-processing" containerID="cri-o://c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815" gracePeriod=30 Oct 29 22:04:56.410176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.410147 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:04:56.568928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.558647 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\" (UniqueName: \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\") pod \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " Oct 29 22:04:56.568928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.558876 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\") pod \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\" (UID: \"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e\") " Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.612115 4376 generic.go:356] "Generic (PLEG): container finished" podID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" containerID="c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815" exitCode=0 Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.615577 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.616009 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" event={"ID":"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e","Type":"ContainerDied","Data":"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815"} Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.616068 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn" event={"ID":"40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e","Type":"ContainerDied","Data":"bb7306beaa78dbdaaf842d010a85f1098f4a6103c9db4a1d8b4e7a022a3f3430"} Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.616098 4376 scope.go:122] "RemoveContainer" containerID="c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815" Oct 29 22:04:56.685000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.677264 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume" pod "40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" (UID: "40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:56.841576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.781484 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.841576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.815399 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd" pod "40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" (UID: "40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e"). InnerVolumeSpecName "kube-api-access-cb5pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:56.865172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.865123 4376 scope.go:122] "RemoveContainer" containerID="c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815" Oct 29 22:04:56.867393 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:04:56.866218 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815\": container with ID starting with c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815 not found: ID does not exist" containerID="c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815" Oct 29 22:04:56.867393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.866258 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815"} err="failed to get container status \"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815\": rpc error: code = NotFound desc = could not find container \"c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815\": container with ID starting with c0d14a44c5d43ce9ebe3251e36c36a92c8c8363c5d391155fe211cf0f8af3815 not found: ID does not exist" Oct 29 22:04:56.884200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.883233 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5pd\" (UniqueName: \"kubernetes.io/projected/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e-kube-api-access-cb5pd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.970505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.970212 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:04:56.980171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:56.980134 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-76f7dbbf8d-pzfpn"] Oct 29 22:04:57.345876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:57.344135 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" path="/var/lib/kubelet/pods/40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e/volumes" Oct 29 22:04:58.327993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.326026 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:04:58.506921 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.503501 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:04:58.647436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.647346 4376 generic.go:356] "Generic (PLEG): container finished" podID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerID="e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c" exitCode=0 Oct 29 22:04:58.647979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.647898 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerDied","Data":"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c"} Oct 29 22:04:58.648668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.648648 4376 scope.go:122] "RemoveContainer" containerID="e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c" Oct 29 22:04:58.651848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.651811 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:58.652069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.652048 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:58.652204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:58.652187 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:04:59.700925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:04:59.688593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerStarted","Data":"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831"} Oct 29 22:05:00.030599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:00.030517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:05:00.031574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:00.031550 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:00.031664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:00.031603 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:01.031276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:01.031243 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:01.031686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:01.031306 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:02.030453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:02.030418 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:02.030659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:02.030478 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:02.402532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:02.402455 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:05:02.689075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:02.688989 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:05:02.689233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:02.689201 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" containerID="cri-o://a422545cf321c68715f0b2519b37323a0b45396e4e27dc21742b16aa57be2a28" gracePeriod=30 Oct 29 22:05:03.031066 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.031032 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:03.031242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.031083 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:03.326598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.326519 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" podUID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" containerName="payload-processing" containerID="cri-o://237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59" gracePeriod=25 Oct 29 22:05:03.504185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.504118 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" podUID="8c741a1d-737c-4568-bc28-69b6a434f51f" containerName="payload-pre-processing" containerID="cri-o://846b8516867e03f812835a97f2483c24fdefa8c69830b45de5c8f36542236dea" gracePeriod=25 Oct 29 22:05:03.740412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.740070 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:05:03.856021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.832268 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\") pod \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " Oct 29 22:05:03.856021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.832598 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\" (UniqueName: \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\") pod \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\" (UID: \"660565e7-f6c3-4b24-939e-b7ad7f095b6b\") " Oct 29 22:05:03.862797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.862109 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume" pod "660565e7-f6c3-4b24-939e-b7ad7f095b6b" (UID: "660565e7-f6c3-4b24-939e-b7ad7f095b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:03.865026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.864983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4" pod "660565e7-f6c3-4b24-939e-b7ad7f095b6b" (UID: "660565e7-f6c3-4b24-939e-b7ad7f095b6b"). InnerVolumeSpecName "kube-api-access-d68z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:03.878352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.878178 4376 generic.go:356] "Generic (PLEG): container finished" podID="8c741a1d-737c-4568-bc28-69b6a434f51f" containerID="846b8516867e03f812835a97f2483c24fdefa8c69830b45de5c8f36542236dea" exitCode=0 Oct 29 22:05:03.878352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.878220 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" event={"ID":"8c741a1d-737c-4568-bc28-69b6a434f51f","Type":"ContainerDied","Data":"846b8516867e03f812835a97f2483c24fdefa8c69830b45de5c8f36542236dea"} Oct 29 22:05:03.878352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.878285 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" event={"ID":"8c741a1d-737c-4568-bc28-69b6a434f51f","Type":"ContainerDied","Data":"04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b"} Oct 29 22:05:03.878352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.878308 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d132949d2a31f032c96cad43a070041a7cee4cac44972515be9f94e031dd7b" Oct 29 22:05:03.885394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.885370 4376 generic.go:356] "Generic (PLEG): container finished" podID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" containerID="237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59" exitCode=0 Oct 29 22:05:03.885513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.885433 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" Oct 29 22:05:03.885572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.885524 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" event={"ID":"660565e7-f6c3-4b24-939e-b7ad7f095b6b","Type":"ContainerDied","Data":"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59"} Oct 29 22:05:03.885572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.885558 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6" event={"ID":"660565e7-f6c3-4b24-939e-b7ad7f095b6b","Type":"ContainerDied","Data":"d0212c5db9b20c4ee76f92fb4ef14d263c6dead62bafab74e37f11345a01af39"} Oct 29 22:05:03.885680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.885582 4376 scope.go:122] "RemoveContainer" containerID="237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59" Oct 29 22:05:03.908032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.908006 4376 scope.go:122] "RemoveContainer" containerID="237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59" Oct 29 22:05:03.908358 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:03.908329 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59\": container with ID starting with 237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59 not found: ID does not exist" containerID="237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59" Oct 29 22:05:03.908448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.908368 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59"} err="failed to get container status \"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59\": rpc error: code = NotFound desc = could not find container \"237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59\": container with ID starting with 237ae42ed0268813ceb07d935818c97b69dd4b1084172b7aeb96ee03998dca59 not found: ID does not exist" Oct 29 22:05:03.911471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.911454 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:05:03.963166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.963135 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68z4\" (UniqueName: \"kubernetes.io/projected/660565e7-f6c3-4b24-939e-b7ad7f095b6b-kube-api-access-d68z4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:03.963166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.963167 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660565e7-f6c3-4b24-939e-b7ad7f095b6b-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:03.971861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.971835 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:05:03.975085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:03.975064 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-296d19-7747cc4c98-495l6"] Oct 29 22:05:04.031202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.031170 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:04.031356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.031230 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:04.065851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.063705 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\" (UniqueName: \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\") pod \"8c741a1d-737c-4568-bc28-69b6a434f51f\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " Oct 29 22:05:04.065851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.064148 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\") pod \"8c741a1d-737c-4568-bc28-69b6a434f51f\" (UID: \"8c741a1d-737c-4568-bc28-69b6a434f51f\") " Oct 29 22:05:04.065851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.065458 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume" pod "8c741a1d-737c-4568-bc28-69b6a434f51f" (UID: "8c741a1d-737c-4568-bc28-69b6a434f51f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:04.068450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.068423 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq" pod "8c741a1d-737c-4568-bc28-69b6a434f51f" (UID: "8c741a1d-737c-4568-bc28-69b6a434f51f"). InnerVolumeSpecName "kube-api-access-m4wzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:04.167782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.167174 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c741a1d-737c-4568-bc28-69b6a434f51f-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.167782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.167198 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wzq\" (UniqueName: \"kubernetes.io/projected/8c741a1d-737c-4568-bc28-69b6a434f51f-kube-api-access-m4wzq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.498997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.498953 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:05:04.584928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.500387 4376 cpu_manager.go:414] "containerMap: removing container" podUID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" containerName="payload-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.720953 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" containerName="payload-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.720997 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8c741a1d-737c-4568-bc28-69b6a434f51f" containerName="payload-pre-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721006 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c741a1d-737c-4568-bc28-69b6a434f51f" containerName="payload-pre-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721048 4376 cpu_manager.go:414] "containerMap: removing container" podUID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" containerName="payload-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721056 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" containerName="payload-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721339 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" containerName="payload-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721351 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c741a1d-737c-4568-bc28-69b6a434f51f" containerName="payload-pre-processing" Oct 29 22:05:04.722636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.721362 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fd94a1-f0c5-4cda-9a30-4c43b08d1d8e" containerName="payload-processing" Oct 29 22:05:04.727762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.727634 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:05:04.728038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.727734 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:04.750060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.745475 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7046db-dockercfg-f5bp7\"" Oct 29 22:05:04.750060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.746266 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7046db\"" Oct 29 22:05:04.884208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.884174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmrx\" (UniqueName: \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:04.884453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.884419 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:04.895742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.895718 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7" Oct 29 22:05:04.963027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.962990 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:05:04.968400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.968369 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-296d19-68d4bf6575-298l7"] Oct 29 22:05:04.985483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.985416 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:04.985662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.985642 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmrx\" (UniqueName: \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:04.986228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:04.986200 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:05.031927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.030457 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:05:05.031927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.030527 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 29 22:05:05.056053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.056018 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmrx\" (UniqueName: \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\") pod \"payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:05.057450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.057423 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:05.301629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.301536 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660565e7-f6c3-4b24-939e-b7ad7f095b6b" path="/var/lib/kubelet/pods/660565e7-f6c3-4b24-939e-b7ad7f095b6b/volumes" Oct 29 22:05:05.302251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.302228 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c741a1d-737c-4568-bc28-69b6a434f51f" path="/var/lib/kubelet/pods/8c741a1d-737c-4568-bc28-69b6a434f51f/volumes" Oct 29 22:05:05.580797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.580764 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:05:05.581478 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:05.581451 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b12b0b_8057_4473_93c4_41349500914b.slice/crio-9fc620b56fa4006bbd7f6372c40d33c6c60b5a07dbf113385aa70acf41871d9a WatchSource:0}: Error finding container 9fc620b56fa4006bbd7f6372c40d33c6c60b5a07dbf113385aa70acf41871d9a: Status 404 returned error can't find the container with id 9fc620b56fa4006bbd7f6372c40d33c6c60b5a07dbf113385aa70acf41871d9a Oct 29 22:05:05.905735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.905651 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" event={"ID":"c2b12b0b-8057-4473-93c4-41349500914b","Type":"ContainerStarted","Data":"4f1767bc48d5f5f5650e2d240d13022dfbf92dc8a947b2390e90435c0b6628ce"} Oct 29 22:05:05.905735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.905704 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" event={"ID":"c2b12b0b-8057-4473-93c4-41349500914b","Type":"ContainerStarted","Data":"9fc620b56fa4006bbd7f6372c40d33c6c60b5a07dbf113385aa70acf41871d9a"} Oct 29 22:05:05.906194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.905780 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:05.958267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:05.958196 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" podStartSLOduration=1.958175577 podStartE2EDuration="1.958175577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:05.951184244 +0000 UTC m=+1364.799661090" watchObservedRunningTime="2026-10-29 22:05:05.958175577 +0000 UTC m=+1364.806652425" Oct 29 22:05:06.065991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.058894 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:05:06.065991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.059425 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:05:06.065991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.061980 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:05:06.403927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.380693 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:05:06.404490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.404446 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" containerID="cri-o://109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4" gracePeriod=30 Oct 29 22:05:06.640926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.635136 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:05:06.666993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:06.665865 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:05:07.951049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:07.951023 4376 generic.go:356] "Generic (PLEG): container finished" podID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerID="a422545cf321c68715f0b2519b37323a0b45396e4e27dc21742b16aa57be2a28" exitCode=0 Oct 29 22:05:07.951371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:07.951077 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" event={"ID":"90ec82d2-aa23-46ac-a103-fa51555c216b","Type":"ContainerDied","Data":"a422545cf321c68715f0b2519b37323a0b45396e4e27dc21742b16aa57be2a28"} Oct 29 22:05:07.951371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:07.951141 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" event={"ID":"90ec82d2-aa23-46ac-a103-fa51555c216b","Type":"ContainerDied","Data":"369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a"} Oct 29 22:05:07.951371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:07.951169 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369bbdb3900c3e593314bc66a92e3d3e9fb57c49e992c5989b984b521070519a" Oct 29 22:05:07.963414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:07.963386 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:05:08.013027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.012997 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013079 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013116 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013147 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013357 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013390 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013443 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:08.013534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013454 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:08.013534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013471 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013503 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013629 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\") pod \"90ec82d2-aa23-46ac-a103-fa51555c216b\" (UID: \"90ec82d2-aa23-46ac-a103-fa51555c216b\") " Oct 29 22:05:08.013799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:08.013799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013755 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:08.013971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.013949 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:08.014513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.014489 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.014619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.014516 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.014619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.014535 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.014619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.014550 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.014619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.014564 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec82d2-aa23-46ac-a103-fa51555c216b-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.015814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.015787 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:08.015814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.015798 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:08.015995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.015809 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:08.016096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.016077 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s" pod "90ec82d2-aa23-46ac-a103-fa51555c216b" (UID: "90ec82d2-aa23-46ac-a103-fa51555c216b"). InnerVolumeSpecName "kube-api-access-zln9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:08.116105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.116071 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.116105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.116100 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zln9s\" (UniqueName: \"kubernetes.io/projected/90ec82d2-aa23-46ac-a103-fa51555c216b-kube-api-access-zln9s\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.116298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.116114 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.116298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.116126 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec82d2-aa23-46ac-a103-fa51555c216b-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:08.168081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.168052 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:05:08.174939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.174892 4376 cpu_manager.go:414] "containerMap: removing container" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" Oct 29 22:05:08.174939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.174938 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" Oct 29 22:05:08.175466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.175446 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" containerName="istio-proxy" Oct 29 22:05:08.181227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.181201 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.197119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.197088 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:05:08.199957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.199931 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b16b28-openshift-default-dockercfg-l4cq8\"" Oct 29 22:05:08.217946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.217843 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.217946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.217923 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218069 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218119 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218197 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218221 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218318 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.218470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.218436 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm9dj\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319466 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319541 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319691 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319748 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319874 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319939 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.319988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.319971 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm9dj\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320160 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320209 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320257 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320741 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.320867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.320801 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.322269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.322247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.322269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.322263 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.346535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.346507 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.346644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.346593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm9dj\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\") pod \"e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.511305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.511211 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:08.954692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.954667 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:05:08.955105 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:08.954776 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f39547_4c22_4c65_8088_7519d945ac7b.slice/crio-fee20615358781c96a8ce0e2d4c8bbada20705cbfe7d93ce27efddcb8647d9ed WatchSource:0}: Error finding container fee20615358781c96a8ce0e2d4c8bbada20705cbfe7d93ce27efddcb8647d9ed: Status 404 returned error can't find the container with id fee20615358781c96a8ce0e2d4c8bbada20705cbfe7d93ce27efddcb8647d9ed Oct 29 22:05:08.957498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.957463 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:08.957619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.957600 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:08.957681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.957655 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:08.960917 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:08.960881 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q" Oct 29 22:05:09.025695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:09.025653 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:05:09.038132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:09.038102 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-l4l9q"] Oct 29 22:05:09.275730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:09.275699 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ec82d2-aa23-46ac-a103-fa51555c216b" path="/var/lib/kubelet/pods/90ec82d2-aa23-46ac-a103-fa51555c216b/volumes" Oct 29 22:05:09.969159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:09.969128 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerStarted","Data":"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2"} Oct 29 22:05:09.969564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:09.969183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerStarted","Data":"fee20615358781c96a8ce0e2d4c8bbada20705cbfe7d93ce27efddcb8647d9ed"} Oct 29 22:05:10.047447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:10.047395 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podStartSLOduration=2.047381444 podStartE2EDuration="2.047381444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:10.031166 +0000 UTC m=+1368.879642847" watchObservedRunningTime="2026-10-29 22:05:10.047381444 +0000 UTC m=+1368.895858287" Oct 29 22:05:10.511871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:10.511837 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:10.517028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:10.517003 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:10.976646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:10.976618 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:10.977688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:10.977667 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:05:11.825415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.825347 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:05:11.901263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901226 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901286 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901422 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901454 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901549 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.901686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901573 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901613 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901670 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.901819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.901795 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\") pod \"cc336c69-f397-4df8-b330-d2945537d933\" (UID: \"cc336c69-f397-4df8-b330-d2945537d933\") " Oct 29 22:05:11.902721 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.902694 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.903064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.903040 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:11.903314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.903291 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.903531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.903510 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.903705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.903687 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.904085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.904059 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "kube-api-access-jqb9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.905618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.905592 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:11.906043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.906018 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.908008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.907987 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy" pod "cc336c69-f397-4df8-b330-d2945537d933" (UID: "cc336c69-f397-4df8-b330-d2945537d933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.985016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.984984 4376 generic.go:356] "Generic (PLEG): container finished" podID="cc336c69-f397-4df8-b330-d2945537d933" containerID="109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4" exitCode=0 Oct 29 22:05:11.985391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.985029 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" Oct 29 22:05:11.985391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.985070 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerDied","Data":"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4"} Oct 29 22:05:11.985391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.985119 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25" event={"ID":"cc336c69-f397-4df8-b330-d2945537d933","Type":"ContainerDied","Data":"0a03238dfa847da579e78e9f3197811e93e7c43418d24b9e6306735d5b670acd"} Oct 29 22:05:11.985391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.985141 4376 scope.go:122] "RemoveContainer" containerID="109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4" Oct 29 22:05:11.996567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:11.996548 4376 scope.go:122] "RemoveContainer" containerID="d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872" Oct 29 22:05:12.005219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005192 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqb9p\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-kube-api-access-jqb9p\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005355 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005454 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc336c69-f397-4df8-b330-d2945537d933-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005534 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc336c69-f397-4df8-b330-d2945537d933-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005602 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005674 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005749 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc336c69-f397-4df8-b330-d2945537d933-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.005831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.005822 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc336c69-f397-4df8-b330-d2945537d933-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:12.016414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.016393 4376 scope.go:122] "RemoveContainer" containerID="109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4" Oct 29 22:05:12.016673 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:12.016654 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4\": container with ID starting with 109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4 not found: ID does not exist" containerID="109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4" Oct 29 22:05:12.016716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.016682 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4"} err="failed to get container status \"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4\": rpc error: code = NotFound desc = could not find container \"109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4\": container with ID starting with 109d17d88c2d328305488c07f9a0d4c81ba57cae4fa30f3deb6fa6ceefa2dec4 not found: ID does not exist" Oct 29 22:05:12.016716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.016700 4376 scope.go:122] "RemoveContainer" containerID="d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872" Oct 29 22:05:12.016942 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:12.016924 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872\": container with ID starting with d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872 not found: ID does not exist" containerID="d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872" Oct 29 22:05:12.017007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.016951 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872"} err="failed to get container status \"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872\": rpc error: code = NotFound desc = could not find container \"d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872\": container with ID starting with d5251e9f602c8862f506552da729d767873ab36dc633e2668183fa5bbf3b2872 not found: ID does not exist" Oct 29 22:05:12.046528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.046504 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:05:12.053179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:12.052953 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fd295-openshift-default-7847778489-77c25"] Oct 29 22:05:13.273462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:13.273431 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc336c69-f397-4df8-b330-d2945537d933" path="/var/lib/kubelet/pods/cc336c69-f397-4df8-b330-d2945537d933/volumes" Oct 29 22:05:15.921926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.913289 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:05:15.921926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.914449 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:15.921926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.914475 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:15.921926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.914671 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:15.940298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.940254 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:15.966927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.942799 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:05:15.967222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.963290 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fe45b\"" Oct 29 22:05:15.983518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:15.983447 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fe45b-dockercfg-s8lbh\"" Oct 29 22:05:16.040267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.040225 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.040649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.040623 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrk5s\" (UniqueName: \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.288155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.277818 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.384446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.384406 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrk5s\" (UniqueName: \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.407604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.347757 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.426277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.346994 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:05:16.428497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.428464 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:16.428686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.428670 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:16.526111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.510669 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrk5s\" (UniqueName: \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.537579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.537045 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc336c69-f397-4df8-b330-d2945537d933" containerName="istio-proxy" Oct 29 22:05:16.596420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.559004 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:05:16.596420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.559147 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.596420 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:16.585350 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:05:16.653547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.648178 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2p8\" (UniqueName: \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.653547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.648296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.653547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.652011 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:16.749329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.749293 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 22:05:16.749844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.749713 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" containerID="cri-o://f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e" gracePeriod=30 Oct 29 22:05:16.750561 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.749958 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2p8\" (UniqueName: \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.750690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.750669 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.751669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.751649 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:16.778020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.777890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2p8\" (UniqueName: \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\") pod \"payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:17.003171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:16.961198 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:05:17.097925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.071199 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:17.256891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.256861 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:05:17.257266 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:17.257240 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639993b7_932e_4bbd_b997_990ceda36487.slice/crio-1ea85f25e95facdf551221781d65033df68b2aaec871c5c68195c1290ec00a14 WatchSource:0}: Error finding container 1ea85f25e95facdf551221781d65033df68b2aaec871c5c68195c1290ec00a14: Status 404 returned error can't find the container with id 1ea85f25e95facdf551221781d65033df68b2aaec871c5c68195c1290ec00a14 Oct 29 22:05:17.490596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.490567 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:05:17.491200 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:17.491176 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a33cc_0d5a_4ff8_a118_3d8d6a71d82a.slice/crio-2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8 WatchSource:0}: Error finding container 2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8: Status 404 returned error can't find the container with id 2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8 Oct 29 22:05:17.543187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.543156 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" event={"ID":"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a","Type":"ContainerStarted","Data":"2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8"} Oct 29 22:05:17.545127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.545091 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" event={"ID":"639993b7-932e-4bbd-b997-990ceda36487","Type":"ContainerStarted","Data":"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55"} Oct 29 22:05:17.545234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.545144 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" event={"ID":"639993b7-932e-4bbd-b997-990ceda36487","Type":"ContainerStarted","Data":"1ea85f25e95facdf551221781d65033df68b2aaec871c5c68195c1290ec00a14"} Oct 29 22:05:17.545377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.545360 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:17.596677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:17.596628 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" podStartSLOduration=2.596604037 podStartE2EDuration="2.596604037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:17.592797167 +0000 UTC m=+1376.441274072" watchObservedRunningTime="2026-10-29 22:05:17.596604037 +0000 UTC m=+1376.445080884" Oct 29 22:05:18.559755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:18.559584 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" event={"ID":"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a","Type":"ContainerStarted","Data":"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee"} Oct 29 22:05:18.560285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:18.560102 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:18.606342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:18.606287 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" podStartSLOduration=2.606272271 podStartE2EDuration="2.606272271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:18.604980744 +0000 UTC m=+1377.453457602" watchObservedRunningTime="2026-10-29 22:05:18.606272271 +0000 UTC m=+1377.454749135" Oct 29 22:05:19.736104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:19.736070 4376 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:05:19.736496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:19.736122 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:05:22.062783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.062752 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:05:22.134226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134189 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134342 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134423 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134521 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134573 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134643 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134691 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.134809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134782 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134817 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.134934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.134886 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\") pod \"2dd67725-79b9-499b-a238-61d27d1db363\" (UID: \"2dd67725-79b9-499b-a238-61d27d1db363\") " Oct 29 22:05:22.135039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135013 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.135153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135129 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.135274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135245 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:22.135325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135283 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.135991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135963 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.136123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.135994 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.136123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.136010 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd67725-79b9-499b-a238-61d27d1db363-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.136123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.136024 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.136123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.136038 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.138363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.137258 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:22.138363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.137614 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:22.138363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.137782 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.138918 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.138871 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn" pod "2dd67725-79b9-499b-a238-61d27d1db363" (UID: "2dd67725-79b9-499b-a238-61d27d1db363"). InnerVolumeSpecName "kube-api-access-6fmxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:22.237960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.237923 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmxn\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-kube-api-access-6fmxn\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.237960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.237955 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd67725-79b9-499b-a238-61d27d1db363-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.237960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.237967 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd67725-79b9-499b-a238-61d27d1db363-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.238210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.237976 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd67725-79b9-499b-a238-61d27d1db363-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.594865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.594825 4376 generic.go:356] "Generic (PLEG): container finished" podID="2dd67725-79b9-499b-a238-61d27d1db363" containerID="f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e" exitCode=0 Oct 29 22:05:22.595092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.594889 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" Oct 29 22:05:22.595092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.594919 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerDied","Data":"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e"} Oct 29 22:05:22.595092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.595081 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd" event={"ID":"2dd67725-79b9-499b-a238-61d27d1db363","Type":"ContainerDied","Data":"0f671fd6a820dd96d7a96a9059e80dfbe68ccc7748614b4027b9484bb39dc2ec"} Oct 29 22:05:22.595257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.595107 4376 scope.go:122] "RemoveContainer" containerID="f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e" Oct 29 22:05:22.609575 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.609546 4376 scope.go:122] "RemoveContainer" containerID="3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23" Oct 29 22:05:22.621361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.621338 4376 scope.go:122] "RemoveContainer" containerID="f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e" Oct 29 22:05:22.621613 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:22.621588 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e\": container with ID starting with f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e not found: ID does not exist" containerID="f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e" Oct 29 22:05:22.621669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.621629 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e"} err="failed to get container status \"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e\": rpc error: code = NotFound desc = could not find container \"f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e\": container with ID starting with f2bec35e6165c30483b69d8917dfedb43fffcc22824be286a0eb32bac9680a8e not found: ID does not exist" Oct 29 22:05:22.621669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.621657 4376 scope.go:122] "RemoveContainer" containerID="3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23" Oct 29 22:05:22.621886 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:22.621862 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23\": container with ID starting with 3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23 not found: ID does not exist" containerID="3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23" Oct 29 22:05:22.621970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.621899 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23"} err="failed to get container status \"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23\": rpc error: code = NotFound desc = could not find container \"3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23\": container with ID starting with 3dbcb1ed132b0c3021e0544694ed8f479b4b19ef892c213d4b9b59268ca66e23 not found: ID does not exist" Oct 29 22:05:22.670609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.670575 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 22:05:22.684032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:22.684001 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-2jbwd"] Oct 29 22:05:23.274845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:23.274803 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd67725-79b9-499b-a238-61d27d1db363" path="/var/lib/kubelet/pods/2dd67725-79b9-499b-a238-61d27d1db363/volumes" Oct 29 22:05:28.562469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:28.562436 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:05:29.161709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.161677 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:05:29.162644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.162619 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:29.162644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.162643 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:29.162761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.162686 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:29.162761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.162692 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:29.162832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.162794 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:29.166449 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.166429 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.206042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.203381 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:05:29.208031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.206436 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7046db-serving-cert\"" Oct 29 22:05:29.218941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.218880 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.219348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.219325 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vjf\" (UniqueName: \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.219494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.219479 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.219626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.219612 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.321286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.321250 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.321487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.321387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vjf\" (UniqueName: \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.321487 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:29.321413 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-serving-cert: secret "maas-api-e2e-praxis-b-7046db-serving-cert" not found Oct 29 22:05:29.321630 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:29.321500 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls podName:0e1e669b-d650-4cfa-8500-7cfa709ee4a4 nodeName:}" failed. No retries permitted until 2026-10-29 22:05:29.821477504 +0000 UTC m=+1388.669954348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls") pod "maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" (UID: "0e1e669b-d650-4cfa-8500-7cfa709ee4a4") : secret "maas-api-e2e-praxis-b-7046db-serving-cert" not found Oct 29 22:05:29.321701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.321621 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.321759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.321700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.322333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.322310 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.324215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.324189 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.365538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.365411 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vjf\" (UniqueName: \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.572937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.572890 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:05:29.836736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.836646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:29.839346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:29.839318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:30.086979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.086583 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:30.534826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.530887 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:05:30.539820 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:30.539792 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1e669b_d650_4cfa_8500_7cfa709ee4a4.slice/crio-71f36633132c3b9242f6b2d9a813f97d8ee11b9f393808510b85a175798f79d8 WatchSource:0}: Error finding container 71f36633132c3b9242f6b2d9a813f97d8ee11b9f393808510b85a175798f79d8: Status 404 returned error can't find the container with id 71f36633132c3b9242f6b2d9a813f97d8ee11b9f393808510b85a175798f79d8 Oct 29 22:05:30.551031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.542938 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:05:30.789195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.789098 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" event={"ID":"0e1e669b-d650-4cfa-8500-7cfa709ee4a4","Type":"ContainerStarted","Data":"71f36633132c3b9242f6b2d9a813f97d8ee11b9f393808510b85a175798f79d8"} Oct 29 22:05:30.975259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.975230 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:05:30.975482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.975457 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" podUID="950d2aef-54a7-444f-8075-3b0a0309c7df" containerName="payload-processing" containerID="cri-o://2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9" gracePeriod=30 Oct 29 22:05:30.995856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.995819 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:05:30.996523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:30.996486 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" podUID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" containerName="payload-pre-processing" containerID="cri-o://932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf" gracePeriod=30 Oct 29 22:05:31.328188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.311334 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:05:31.398173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.392678 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\") pod \"950d2aef-54a7-444f-8075-3b0a0309c7df\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " Oct 29 22:05:31.401926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.398680 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\" (UniqueName: \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\") pod \"950d2aef-54a7-444f-8075-3b0a0309c7df\" (UID: \"950d2aef-54a7-444f-8075-3b0a0309c7df\") " Oct 29 22:05:31.404738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.404702 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume" pod "950d2aef-54a7-444f-8075-3b0a0309c7df" (UID: "950d2aef-54a7-444f-8075-3b0a0309c7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:31.474200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.474138 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx" pod "950d2aef-54a7-444f-8075-3b0a0309c7df" (UID: "950d2aef-54a7-444f-8075-3b0a0309c7df"). InnerVolumeSpecName "kube-api-access-88ngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:31.503887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.503545 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:05:31.508485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.508414 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ngx\" (UniqueName: \"kubernetes.io/projected/950d2aef-54a7-444f-8075-3b0a0309c7df-kube-api-access-88ngx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.508485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.508445 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950d2aef-54a7-444f-8075-3b0a0309c7df-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.725887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.725138 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\" (UniqueName: \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\") pod \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " Oct 29 22:05:31.725887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.725363 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\") pod \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\" (UID: \"fa1b280d-39a9-406d-afa4-7bbbf7896a28\") " Oct 29 22:05:31.728693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.726394 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume" pod "fa1b280d-39a9-406d-afa4-7bbbf7896a28" (UID: "fa1b280d-39a9-406d-afa4-7bbbf7896a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:31.739519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.739428 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd" pod "fa1b280d-39a9-406d-afa4-7bbbf7896a28" (UID: "fa1b280d-39a9-406d-afa4-7bbbf7896a28"). InnerVolumeSpecName "kube-api-access-l7vcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:31.841929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.837344 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7vcd\" (UniqueName: \"kubernetes.io/projected/fa1b280d-39a9-406d-afa4-7bbbf7896a28-kube-api-access-l7vcd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.841929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.837382 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1b280d-39a9-406d-afa4-7bbbf7896a28-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.928535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.928496 4376 generic.go:356] "Generic (PLEG): container finished" podID="950d2aef-54a7-444f-8075-3b0a0309c7df" containerID="2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9" exitCode=0 Oct 29 22:05:31.928705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.928633 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" event={"ID":"950d2aef-54a7-444f-8075-3b0a0309c7df","Type":"ContainerDied","Data":"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9"} Oct 29 22:05:31.928705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.928673 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" event={"ID":"950d2aef-54a7-444f-8075-3b0a0309c7df","Type":"ContainerDied","Data":"7e8664ee2eafe21d324377686d2e6ced79272aac8e6b8af5afc66d857a64cf3d"} Oct 29 22:05:31.928705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.928693 4376 scope.go:122] "RemoveContainer" containerID="2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9" Oct 29 22:05:31.928892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.928744 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x" Oct 29 22:05:31.931329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.931303 4376 generic.go:356] "Generic (PLEG): container finished" podID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" containerID="932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf" exitCode=0 Oct 29 22:05:31.931483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.931404 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" event={"ID":"fa1b280d-39a9-406d-afa4-7bbbf7896a28","Type":"ContainerDied","Data":"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf"} Oct 29 22:05:31.931483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.931433 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" event={"ID":"fa1b280d-39a9-406d-afa4-7bbbf7896a28","Type":"ContainerDied","Data":"24f79fb3b4adb18e50893d40d7a27385b8191299c48e934c8677e334106549ba"} Oct 29 22:05:31.931600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.931487 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj" Oct 29 22:05:31.949234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.949206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" event={"ID":"0e1e669b-d650-4cfa-8500-7cfa709ee4a4","Type":"ContainerStarted","Data":"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f"} Oct 29 22:05:31.950807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.950784 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:31.951298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.951279 4376 scope.go:122] "RemoveContainer" containerID="2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9" Oct 29 22:05:31.951768 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:31.951741 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9\": container with ID starting with 2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9 not found: ID does not exist" containerID="2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9" Oct 29 22:05:31.951937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.951885 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9"} err="failed to get container status \"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9\": rpc error: code = NotFound desc = could not find container \"2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9\": container with ID starting with 2412eac6ea93fa9349899a227a32e38072125c329cf0c2bb4ff68344a7b636a9 not found: ID does not exist" Oct 29 22:05:31.952048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.952034 4376 scope.go:122] "RemoveContainer" containerID="932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf" Oct 29 22:05:31.964329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.964310 4376 scope.go:122] "RemoveContainer" containerID="932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf" Oct 29 22:05:31.964645 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:31.964622 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf\": container with ID starting with 932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf not found: ID does not exist" containerID="932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf" Oct 29 22:05:31.964756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:31.964653 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf"} err="failed to get container status \"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf\": rpc error: code = NotFound desc = could not find container \"932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf\": container with ID starting with 932022a87c81d0311546e2a537e03c481d280251df6039a78ce7e19453e4d9cf not found: ID does not exist" Oct 29 22:05:32.019004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:32.018878 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" podStartSLOduration=2.618564722 podStartE2EDuration="3.018863804s" totalImagesPullingTime="400.299082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:32.014236168 +0000 UTC m=+1390.862713008" watchObservedRunningTime="2026-10-29 22:05:32.018863804 +0000 UTC m=+1390.867340650" Oct 29 22:05:32.058619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:32.058585 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:05:32.062228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:32.062203 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9f8c0-665b59747b-hq76x"] Oct 29 22:05:32.101142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:32.101112 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:05:32.106541 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:32.106512 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9f8c0-6cd76598cf-bhzwj"] Oct 29 22:05:33.274061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:33.274025 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950d2aef-54a7-444f-8075-3b0a0309c7df" path="/var/lib/kubelet/pods/950d2aef-54a7-444f-8075-3b0a0309c7df/volumes" Oct 29 22:05:33.274489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:33.274474 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" path="/var/lib/kubelet/pods/fa1b280d-39a9-406d-afa4-7bbbf7896a28/volumes" Oct 29 22:05:35.473462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.473366 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:05:35.474610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474579 4376 cpu_manager.go:414] "containerMap: removing container" podUID="950d2aef-54a7-444f-8075-3b0a0309c7df" containerName="payload-processing" Oct 29 22:05:35.474610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474606 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d2aef-54a7-444f-8075-3b0a0309c7df" containerName="payload-processing" Oct 29 22:05:35.474759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474628 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" containerName="payload-pre-processing" Oct 29 22:05:35.474759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474637 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" containerName="payload-pre-processing" Oct 29 22:05:35.474843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474812 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd67725-79b9-499b-a238-61d27d1db363" containerName="istio-proxy" Oct 29 22:05:35.474843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474827 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d2aef-54a7-444f-8075-3b0a0309c7df" containerName="payload-processing" Oct 29 22:05:35.474843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.474841 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1b280d-39a9-406d-afa4-7bbbf7896a28" containerName="payload-pre-processing" Oct 29 22:05:35.480807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.480784 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.500161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.500140 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a20be1-openshift-default-dockercfg-9gnlc\"" Oct 29 22:05:35.532503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.532476 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624367 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624436 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624587 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624636 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624675 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624740 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96sd\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624810 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.627928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.624877 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.740895 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96sd\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.740999 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741056 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741237 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741268 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.741976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.741494 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.742660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.742018 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.855842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.781245 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.855842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.782150 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.855842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.786702 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.855842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.795001 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.873230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.870959 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.882970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.878267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96sd\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.920621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.913752 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:35.931940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:35.929432 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\") pod \"e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:36.096458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:36.096378 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:36.422092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:36.422005 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:05:36.434347 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:36.434308 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2097b8_6e8f_4e66_be1a_0aee6587d7d6.slice/crio-f2dc0e1ae4b50c807470003f5edf62be71aac85e731f45135770504f17451b3f WatchSource:0}: Error finding container f2dc0e1ae4b50c807470003f5edf62be71aac85e731f45135770504f17451b3f: Status 404 returned error can't find the container with id f2dc0e1ae4b50c807470003f5edf62be71aac85e731f45135770504f17451b3f Oct 29 22:05:36.437156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:36.437121 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:36.437240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:36.437195 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:36.437240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:36.437227 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:37.037084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:37.037043 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerStarted","Data":"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6"} Oct 29 22:05:37.037505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:37.037105 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerStarted","Data":"f2dc0e1ae4b50c807470003f5edf62be71aac85e731f45135770504f17451b3f"} Oct 29 22:05:37.119929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:37.110291 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:37.169974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:37.161257 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:37.239441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:37.239346 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podStartSLOduration=2.239323568 podStartE2EDuration="2.239323568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:37.110664364 +0000 UTC m=+1395.959141209" watchObservedRunningTime="2026-10-29 22:05:37.239323568 +0000 UTC m=+1396.087800417" Oct 29 22:05:38.044562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:38.044525 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:38.045712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:38.045689 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:05:38.971580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:38.971544 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:05:39.560472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.560436 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:05:39.565434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.565413 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.603105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.603071 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b16b28-serving-cert\"" Oct 29 22:05:39.603354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.603333 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:05:39.611120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.611087 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.611319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.611288 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.611453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.611356 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.611453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.611408 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8l6\" (UniqueName: \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.712720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.712684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.712935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.712827 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.712935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.712865 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.712935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.712923 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8l6\" (UniqueName: \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.713520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.713497 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.715568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.715542 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.715736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.715611 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.738778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.738754 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8l6\" (UniqueName: \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\") pod \"maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:39.882962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:39.881197 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:40.197571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:40.197540 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:05:40.210372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:40.210332 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" containerID="cri-o://db53580a52d570447057f23f41fde29f7f7e95cb03c035f434f9960e3ee2c309" gracePeriod=30 Oct 29 22:05:40.805975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:40.805938 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:05:40.806766 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:40.806740 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adb92bb_36c8_41b8_a07e_d2995bda710b.slice/crio-f9f42059cc89a8f3299bce006a5a9f4a767a028b6cd9da17d3ff11c2967ca7f8 WatchSource:0}: Error finding container f9f42059cc89a8f3299bce006a5a9f4a767a028b6cd9da17d3ff11c2967ca7f8: Status 404 returned error can't find the container with id f9f42059cc89a8f3299bce006a5a9f4a767a028b6cd9da17d3ff11c2967ca7f8 Oct 29 22:05:41.324181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:41.324153 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" event={"ID":"1adb92bb-36c8-41b8-a07e-d2995bda710b","Type":"ContainerStarted","Data":"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c"} Oct 29 22:05:41.324320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:41.324203 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" event={"ID":"1adb92bb-36c8-41b8-a07e-d2995bda710b","Type":"ContainerStarted","Data":"f9f42059cc89a8f3299bce006a5a9f4a767a028b6cd9da17d3ff11c2967ca7f8"} Oct 29 22:05:41.324320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:41.324255 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:41.373801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:41.373748 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" podStartSLOduration=1.9852873930000001 podStartE2EDuration="2.373735001s" totalImagesPullingTime="388.447608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:41.369756366 +0000 UTC m=+1400.218233212" watchObservedRunningTime="2026-10-29 22:05:41.373735001 +0000 UTC m=+1400.222211847" Oct 29 22:05:45.429220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:45.429184 4376 generic.go:356] "Generic (PLEG): container finished" podID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerID="db53580a52d570447057f23f41fde29f7f7e95cb03c035f434f9960e3ee2c309" exitCode=0 Oct 29 22:05:45.429682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:45.429248 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerDied","Data":"db53580a52d570447057f23f41fde29f7f7e95cb03c035f434f9960e3ee2c309"} Oct 29 22:05:45.429682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:45.429308 4376 scope.go:122] "RemoveContainer" containerID="e7ca71e58a9d5c412bc781f96652d426fecea53fe809879673e8ee1005ec71be" Oct 29 22:05:46.047984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.045391 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143398 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143552 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143621 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143663 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143714 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143788 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.143931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.143941 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.144682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.144014 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.144682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.144079 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\") pod \"8a97ab11-38d6-40a5-9658-3b49c59b98db\" (UID: \"8a97ab11-38d6-40a5-9658-3b49c59b98db\") " Oct 29 22:05:46.164692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.146510 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:46.164692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.147047 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:46.164692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.148213 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:46.164692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.148641 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:46.164692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.148788 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:46.165685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.165634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "kube-api-access-kksbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:46.166116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.166088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:46.171840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.171807 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:46.172000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.171950 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy" pod "8a97ab11-38d6-40a5-9658-3b49c59b98db" (UID: "8a97ab11-38d6-40a5-9658-3b49c59b98db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287499 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287542 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287556 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287568 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287583 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kksbj\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-kube-api-access-kksbj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287599 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287613 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a97ab11-38d6-40a5-9658-3b49c59b98db-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287627 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a97ab11-38d6-40a5-9658-3b49c59b98db-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.287927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.287660 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a97ab11-38d6-40a5-9658-3b49c59b98db-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.305001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.304676 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:05:46.306891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.305290 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" containerID="cri-o://75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160" gracePeriod=30 Oct 29 22:05:46.313086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.313056 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:05:46.313313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.313286 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" podUID="c321d293-6700-43ac-be55-6b1f5d067ca7" containerName="maas-api" containerID="cri-o://143e631ebe8287404226face3de67a7edfe953020d4d99ebe10c00652c2848d3" gracePeriod=30 Oct 29 22:05:46.439818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.439786 4376 generic.go:356] "Generic (PLEG): container finished" podID="c321d293-6700-43ac-be55-6b1f5d067ca7" containerID="143e631ebe8287404226face3de67a7edfe953020d4d99ebe10c00652c2848d3" exitCode=0 Oct 29 22:05:46.440250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.439854 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" event={"ID":"c321d293-6700-43ac-be55-6b1f5d067ca7","Type":"ContainerDied","Data":"143e631ebe8287404226face3de67a7edfe953020d4d99ebe10c00652c2848d3"} Oct 29 22:05:46.441485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.441466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" Oct 29 22:05:46.441600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.441498 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn" event={"ID":"8a97ab11-38d6-40a5-9658-3b49c59b98db","Type":"ContainerDied","Data":"7025e8dbb708ec1a8d2e3d806267f6a60d7b6cddb5a139d8369d842c2892c4a6"} Oct 29 22:05:46.441600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.441557 4376 scope.go:122] "RemoveContainer" containerID="db53580a52d570447057f23f41fde29f7f7e95cb03c035f434f9960e3ee2c309" Oct 29 22:05:46.442813 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:46.442793 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:05:46.484348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.484307 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:05:46.484560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.484541 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" podUID="adf52120-be10-4efb-99a4-9d18f6e70dbc" containerName="payload-pre-processing" containerID="cri-o://9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae" gracePeriod=30 Oct 29 22:05:46.515015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.514884 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:05:46.518791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.518763 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9f8c0-openshift-default-5d97f887d5-sl2fn"] Oct 29 22:05:46.657635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.657600 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:05:46.777673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.777644 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:05:46.796598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.796570 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\") pod \"c321d293-6700-43ac-be55-6b1f5d067ca7\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " Oct 29 22:05:46.796745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.796612 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\") pod \"c321d293-6700-43ac-be55-6b1f5d067ca7\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " Oct 29 22:05:46.796745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.796694 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\") pod \"c321d293-6700-43ac-be55-6b1f5d067ca7\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " Oct 29 22:05:46.799779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.799740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca" pod "c321d293-6700-43ac-be55-6b1f5d067ca7" (UID: "c321d293-6700-43ac-be55-6b1f5d067ca7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:46.802007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.801977 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\" (UniqueName: \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\") pod \"c321d293-6700-43ac-be55-6b1f5d067ca7\" (UID: \"c321d293-6700-43ac-be55-6b1f5d067ca7\") " Oct 29 22:05:46.804746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.804717 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c321d293-6700-43ac-be55-6b1f5d067ca7-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.808990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.807757 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs" pod "c321d293-6700-43ac-be55-6b1f5d067ca7" (UID: "c321d293-6700-43ac-be55-6b1f5d067ca7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:46.816846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.816321 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn" pod "c321d293-6700-43ac-be55-6b1f5d067ca7" (UID: "c321d293-6700-43ac-be55-6b1f5d067ca7"). InnerVolumeSpecName "kube-api-access-rsczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:46.817137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.817112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls" pod "c321d293-6700-43ac-be55-6b1f5d067ca7" (UID: "c321d293-6700-43ac-be55-6b1f5d067ca7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:46.907987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.907580 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\") pod \"adf52120-be10-4efb-99a4-9d18f6e70dbc\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " Oct 29 22:05:46.907987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.907729 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\" (UniqueName: \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\") pod \"adf52120-be10-4efb-99a4-9d18f6e70dbc\" (UID: \"adf52120-be10-4efb-99a4-9d18f6e70dbc\") " Oct 29 22:05:46.956059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.908509 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsczn\" (UniqueName: \"kubernetes.io/projected/c321d293-6700-43ac-be55-6b1f5d067ca7-kube-api-access-rsczn\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.956059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.908539 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.956059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:46.908554 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c321d293-6700-43ac-be55-6b1f5d067ca7-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:47.022186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.022144 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume" pod "adf52120-be10-4efb-99a4-9d18f6e70dbc" (UID: "adf52120-be10-4efb-99a4-9d18f6e70dbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:47.037926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.026892 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf52120-be10-4efb-99a4-9d18f6e70dbc-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:47.132937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.132059 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d" pod "adf52120-be10-4efb-99a4-9d18f6e70dbc" (UID: "adf52120-be10-4efb-99a4-9d18f6e70dbc"). InnerVolumeSpecName "kube-api-access-kcz6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:47.142089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.139795 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcz6d\" (UniqueName: \"kubernetes.io/projected/adf52120-be10-4efb-99a4-9d18f6e70dbc-kube-api-access-kcz6d\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:47.276077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.276033 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" path="/var/lib/kubelet/pods/8a97ab11-38d6-40a5-9658-3b49c59b98db/volumes" Oct 29 22:05:47.471931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.452345 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:05:47.523965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.523923 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" event={"ID":"c321d293-6700-43ac-be55-6b1f5d067ca7","Type":"ContainerDied","Data":"6397c8793d1cba4c82e4a4d22f2bfb460983b49e4db5e3cab76a21eac188b3df"} Oct 29 22:05:47.540352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.540271 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" event={"ID":"adf52120-be10-4efb-99a4-9d18f6e70dbc","Type":"ContainerDied","Data":"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae"} Oct 29 22:05:47.540544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.526320 4376 generic.go:356] "Generic (PLEG): container finished" podID="adf52120-be10-4efb-99a4-9d18f6e70dbc" containerID="9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae" exitCode=0 Oct 29 22:05:47.540708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.524260 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz" Oct 29 22:05:47.540981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.526405 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" Oct 29 22:05:47.545936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.541087 4376 scope.go:122] "RemoveContainer" containerID="143e631ebe8287404226face3de67a7edfe953020d4d99ebe10c00652c2848d3" Oct 29 22:05:47.545936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.541221 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb" event={"ID":"adf52120-be10-4efb-99a4-9d18f6e70dbc","Type":"ContainerDied","Data":"99a7e9ab24f28dd46a398b96203c31f0faf740ece58a1a8caea8fd084448993d"} Oct 29 22:05:47.701326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.701004 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:05:47.702680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.702607 4376 patch_prober.go:29] interesting pod/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:05:47.702680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.702660 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:05:47.704869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.704844 4376 scope.go:122] "RemoveContainer" containerID="9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae" Oct 29 22:05:47.709565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.709538 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-85979cbbc9-hf7tb"] Oct 29 22:05:47.722233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.721712 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:05:47.722980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.722606 4376 scope.go:122] "RemoveContainer" containerID="9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae" Oct 29 22:05:47.723080 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:47.723039 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae\": container with ID starting with 9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae not found: ID does not exist" containerID="9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae" Oct 29 22:05:47.723131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.723072 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae"} err="failed to get container status \"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae\": rpc error: code = NotFound desc = could not find container \"9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae\": container with ID starting with 9f4660b4d36303e191cff38a47d09cf09f734aa535f5dfec62a3cdea600d95ae not found: ID does not exist" Oct 29 22:05:47.728088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:47.727350 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad8e69-57fb96b9d-wx2sz"] Oct 29 22:05:48.085114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.084876 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.090926 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c321d293-6700-43ac-be55-6b1f5d067ca7" containerName="maas-api" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.090953 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c321d293-6700-43ac-be55-6b1f5d067ca7" containerName="maas-api" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.091069 4376 cpu_manager.go:414] "containerMap: removing container" podUID="adf52120-be10-4efb-99a4-9d18f6e70dbc" containerName="payload-pre-processing" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.091078 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf52120-be10-4efb-99a4-9d18f6e70dbc" containerName="payload-pre-processing" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.091476 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.091486 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.094030 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.094173 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.094204 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c321d293-6700-43ac-be55-6b1f5d067ca7" containerName="maas-api" Oct 29 22:05:48.096929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.094221 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf52120-be10-4efb-99a4-9d18f6e70dbc" containerName="payload-pre-processing" Oct 29 22:05:48.105567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.105541 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.109694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.109592 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:05:48.124348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.124319 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fd1d4e-openshift-default-dockercfg-jpd57\"" Oct 29 22:05:48.126923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.126882 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zj2q\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.126930 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127083 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127117 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127345 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127385 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127509 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.127587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.127572 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244081 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244184 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244242 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244296 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zj2q\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244328 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244516 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244628 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.244760 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.245358 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.245629 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.247926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.245853 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.249427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.249396 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.271929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.258339 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.271929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.261783 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.272696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.272664 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.292936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.292233 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zj2q\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.296680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.296299 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\") pod \"e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.423747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.423663 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:48.589965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.589935 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:05:48.590696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.590678 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.590746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.590697 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a97ab11-38d6-40a5-9658-3b49c59b98db" containerName="istio-proxy" Oct 29 22:05:48.595760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.595742 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:48.613496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.613474 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9bb81b\"" Oct 29 22:05:48.613641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.613503 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9bb81b-dockercfg-d7nc5\"" Oct 29 22:05:48.616803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.616782 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:05:48.657828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.656428 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:48.657828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.656576 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4s5\" (UniqueName: \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:48.731975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.701273 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:05:48.780933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.770740 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:48.780933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.770869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4s5\" (UniqueName: \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:48.791928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.789115 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:05:48.791928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.789258 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:48.924571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:48.865699 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4s5\" (UniqueName: \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:49.010848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.001318 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.018998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.018944 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m672x\" (UniqueName: \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.026264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.022014 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:49.070864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.070800 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:49.071036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.070990 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:49.071136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.071111 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:05:49.098958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.078376 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:05:49.259027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.223709 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:49.259027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.224252 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.259027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.224433 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m672x\" (UniqueName: \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.259027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.226085 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.428702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.428566 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m672x\" (UniqueName: \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\") pod \"payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.547252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.544169 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf52120-be10-4efb-99a4-9d18f6e70dbc" path="/var/lib/kubelet/pods/adf52120-be10-4efb-99a4-9d18f6e70dbc/volumes" Oct 29 22:05:49.547252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.545151 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c321d293-6700-43ac-be55-6b1f5d067ca7" path="/var/lib/kubelet/pods/c321d293-6700-43ac-be55-6b1f5d067ca7/volumes" Oct 29 22:05:49.673939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.655583 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:49.699133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.699093 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerStarted","Data":"384b97292ef70858b35b619084611d5ab92f3eeb4b106ca7537a70b4fb19b4d8"} Oct 29 22:05:49.712540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.712480 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:05:49.722215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.722122 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:49.749708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.748869 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:05:49.764289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.763761 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5bc531-serving-cert\"" Oct 29 22:05:49.862339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.859823 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:49.862339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.859998 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht75z\" (UniqueName: \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:49.862339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.860145 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:49.862339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.860263 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.035152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.997291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.035152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.997427 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht75z\" (UniqueName: \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.035152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.997597 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.035152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:49.997741 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.035152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.008113 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.138082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.137974 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht75z\" (UniqueName: \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.138579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.138552 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.142292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.142265 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\") pod \"maas-api-e2e-api-a-5bc531-745d867c9b-glct6\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.244017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.243988 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:05:50.245737 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:50.245709 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eca3d2_8cfb_413a_bf90_d0b554072dc3.slice/crio-7e886cf554bcae984f247c6a4821721b3c4b7403b9178fe4aa0f883e831868f1 WatchSource:0}: Error finding container 7e886cf554bcae984f247c6a4821721b3c4b7403b9178fe4aa0f883e831868f1: Status 404 returned error can't find the container with id 7e886cf554bcae984f247c6a4821721b3c4b7403b9178fe4aa0f883e831868f1 Oct 29 22:05:50.280609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.280581 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:05:50.285314 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:50.285287 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a035a3_d117_4936_9fb0_9f3ed354d32a.slice/crio-0355745b3bda62e6e94ccfa58945e8c8e06b31ed2956359889f2998ac81d0fbf WatchSource:0}: Error finding container 0355745b3bda62e6e94ccfa58945e8c8e06b31ed2956359889f2998ac81d0fbf: Status 404 returned error can't find the container with id 0355745b3bda62e6e94ccfa58945e8c8e06b31ed2956359889f2998ac81d0fbf Oct 29 22:05:50.405469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.405399 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:50.708709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.708678 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerStarted","Data":"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc"} Oct 29 22:05:50.710577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.710545 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" event={"ID":"96a035a3-d117-4936-9fb0-9f3ed354d32a","Type":"ContainerStarted","Data":"7293e31127a9490d0c2c702a4eba744eda523cefca7eed82cc596686fd62eb5d"} Oct 29 22:05:50.710721 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.710595 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" event={"ID":"96a035a3-d117-4936-9fb0-9f3ed354d32a","Type":"ContainerStarted","Data":"0355745b3bda62e6e94ccfa58945e8c8e06b31ed2956359889f2998ac81d0fbf"} Oct 29 22:05:50.710721 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.710626 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:05:50.712358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.712334 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" event={"ID":"88eca3d2-8cfb-413a-bf90-d0b554072dc3","Type":"ContainerStarted","Data":"125c7d9a362ee5c0a79584f06261efb7fbdf0e0a30665b194d94ccec04709cf4"} Oct 29 22:05:50.712465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.712375 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" event={"ID":"88eca3d2-8cfb-413a-bf90-d0b554072dc3","Type":"ContainerStarted","Data":"7e886cf554bcae984f247c6a4821721b3c4b7403b9178fe4aa0f883e831868f1"} Oct 29 22:05:50.712465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.712395 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:05:50.717594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.717574 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:05:50.722986 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:05:50.722957 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d92849c_18bc_4694_b787_7633269ba786.slice/crio-7d0e5af76b6ea2d7e60420be5b6fea978a60e3cf7750302ff2f45edebc8c99cd WatchSource:0}: Error finding container 7d0e5af76b6ea2d7e60420be5b6fea978a60e3cf7750302ff2f45edebc8c99cd: Status 404 returned error can't find the container with id 7d0e5af76b6ea2d7e60420be5b6fea978a60e3cf7750302ff2f45edebc8c99cd Oct 29 22:05:50.760139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.760093 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podStartSLOduration=2.760079537 podStartE2EDuration="2.760079537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:50.757038304 +0000 UTC m=+1409.605515163" watchObservedRunningTime="2026-10-29 22:05:50.760079537 +0000 UTC m=+1409.608556383" Oct 29 22:05:50.805035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.804989 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" podStartSLOduration=2.804976428 podStartE2EDuration="2.804976428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:50.800862396 +0000 UTC m=+1409.649339242" watchObservedRunningTime="2026-10-29 22:05:50.804976428 +0000 UTC m=+1409.653453273" Oct 29 22:05:50.849340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:50.849289 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" podStartSLOduration=2.849276639 podStartE2EDuration="2.849276639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:50.84525563 +0000 UTC m=+1409.693732479" watchObservedRunningTime="2026-10-29 22:05:50.849276639 +0000 UTC m=+1409.697753512" Oct 29 22:05:51.424828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.424801 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:51.431823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.431798 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:51.613183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.613160 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:05:51.721467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.721425 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" event={"ID":"6d92849c-18bc-4694-b787-7633269ba786","Type":"ContainerStarted","Data":"c8b1d490e617fe139d89418d72c0f036bff4926e3bdb13a05e3431735b3c7bfb"} Oct 29 22:05:51.721958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.721506 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" event={"ID":"6d92849c-18bc-4694-b787-7633269ba786","Type":"ContainerStarted","Data":"7d0e5af76b6ea2d7e60420be5b6fea978a60e3cf7750302ff2f45edebc8c99cd"} Oct 29 22:05:51.721958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.721539 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:51.723424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723396 4376 generic.go:356] "Generic (PLEG): container finished" podID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerID="75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160" exitCode=0 Oct 29 22:05:51.723525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723431 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" Oct 29 22:05:51.723525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723476 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerDied","Data":"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160"} Oct 29 22:05:51.723525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723522 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k" event={"ID":"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd","Type":"ContainerDied","Data":"908ace16ad3a645198662246633c10258745c4aaa1c3b1b5d6eb226356c481ca"} Oct 29 22:05:51.723695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723552 4376 scope.go:122] "RemoveContainer" containerID="75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160" Oct 29 22:05:51.723798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.723780 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:51.725118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.725101 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:05:51.734520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.734500 4376 scope.go:122] "RemoveContainer" containerID="c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984" Oct 29 22:05:51.745016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.744997 4376 scope.go:122] "RemoveContainer" containerID="75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160" Oct 29 22:05:51.745284 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:51.745264 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160\": container with ID starting with 75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160 not found: ID does not exist" containerID="75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160" Oct 29 22:05:51.745364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.745297 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160"} err="failed to get container status \"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160\": rpc error: code = NotFound desc = could not find container \"75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160\": container with ID starting with 75374dbd34a05f27f863c035afa51884bacc118604d74fe0b7f0f4fc87ed2160 not found: ID does not exist" Oct 29 22:05:51.745364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.745319 4376 scope.go:122] "RemoveContainer" containerID="c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984" Oct 29 22:05:51.745619 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:05:51.745593 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984\": container with ID starting with c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984 not found: ID does not exist" containerID="c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984" Oct 29 22:05:51.745663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.745619 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984"} err="failed to get container status \"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984\": rpc error: code = NotFound desc = could not find container \"c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984\": container with ID starting with c5ca28f3479d64d53c1e1fd2470bd1a6a61cb6ce9471c38bb0147d0d8c4f9984 not found: ID does not exist" Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776438 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776518 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776576 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776623 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776662 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.777925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.776783 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.789033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.779616 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" podStartSLOduration=2.307594129 podStartE2EDuration="2.779592519s" totalImagesPullingTime="471.99839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:51.773240852 +0000 UTC m=+1410.621717699" watchObservedRunningTime="2026-10-29 22:05:51.779592519 +0000 UTC m=+1410.628069366" Oct 29 22:05:51.801166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.798451 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:51.801449 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.798707 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:51.801585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.800118 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:51.801718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.801082 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:51.802094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.802063 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:51.802279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.802248 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.802460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.802441 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\") pod \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\" (UID: \"aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd\") " Oct 29 22:05:51.803829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.803775 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.803983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.803958 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.804154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.804140 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.804233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.804221 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.804303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.804295 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.823926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.822060 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:51.863038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.862995 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:51.864134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.864077 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:51.920201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.900790 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4" pod "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" (UID: "aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd"). InnerVolumeSpecName "kube-api-access-v9xz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:51.920201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.906414 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.920201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.906443 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xz4\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-kube-api-access-v9xz4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.920201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.906460 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:51.920201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:51.906474 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:52.111473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:52.111443 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:05:52.116648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:52.116621 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-296d19-openshift-default-5b744f5b97-6c77k"] Oct 29 22:05:53.273795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:53.273763 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" path="/var/lib/kubelet/pods/aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd/volumes" Oct 29 22:05:57.748428 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:57.748393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:05:59.230314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.230278 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:05:59.231416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.231340 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:05:59.231416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.231364 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:05:59.231618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.231556 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:05:59.237358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.237340 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.262661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.262628 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:05:59.267581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.267549 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-796d71-openshift-default-dockercfg-rrwfs\"" Oct 29 22:05:59.380788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.380750 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.382451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.380807 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.382451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.380937 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxlz\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.382451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.381466 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.382451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.381528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.391794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.391752 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.392265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.392232 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.392614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.392595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.392917 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.392824 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.510079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.509979 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.510411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.510389 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.510615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.510598 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.510822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.510806 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.517930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.510969 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.528065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.528025 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxlz\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.528742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.527435 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.528742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.527923 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.528946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.527636 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.529239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.529219 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.529971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.529858 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.530113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.530093 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.534954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.530545 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.535262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.535244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.543248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.543226 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.585928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.583280 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxlz\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.610927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.608363 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.610927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.608795 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:05:59.892049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:05:59.876229 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:06:00.225394 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:06:00.225362 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789c0500_3aab_4019_bc74_835b516fd5eb.slice/crio-0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154 WatchSource:0}: Error finding container 0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154: Status 404 returned error can't find the container with id 0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154 Oct 29 22:06:00.225606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.225580 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:06:00.228384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.228349 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:00.228473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.228414 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:00.228473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.228446 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:00.890797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.890755 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerStarted","Data":"cfc04ac62d80faa7c6fb85d234916ab259f5eed19a6412ce56214bbcf031f4a7"} Oct 29 22:06:00.891309 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.890825 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerStarted","Data":"0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154"} Oct 29 22:06:00.969214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:00.969160 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podStartSLOduration=1.9691423380000002 podStartE2EDuration="1.969142338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:00.953089657 +0000 UTC m=+1419.801566516" watchObservedRunningTime="2026-10-29 22:06:00.969142338 +0000 UTC m=+1419.817619187" Oct 29 22:06:01.787967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.787152 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:06:01.789095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.789070 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:06:01.878073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.878038 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:06:01.886400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.886375 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:06:01.901640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.901609 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:06:01.903110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:01.903086 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:06:09.050846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.050814 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:06:09.051569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.051553 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:06:09.051622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.051571 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:06:09.051702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.051693 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf57fbd-1cca-4e9d-b3fe-05b24a88a6dd" containerName="istio-proxy" Oct 29 22:06:09.056383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.056365 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.087928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.084599 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:06:09.087928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.085799 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a20be1-serving-cert\"" Oct 29 22:06:09.191192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.191161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.191356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.191334 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrd2g\" (UniqueName: \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.191455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.191441 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.191535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.191525 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.292978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.292945 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrd2g\" (UniqueName: \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.293146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.293024 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.293146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.293105 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.293250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.293158 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.293849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.293825 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.295553 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.295524 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.295690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.295667 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.353930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.353829 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrd2g\" (UniqueName: \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\") pod \"maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.377290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.377247 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:09.761789 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:06:09.761747 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b3bdef_d3ca_4458_bfe3_b708e1a4eab4.slice/crio-9984707238c3ba711cf926b702d59e1340a6e545bce953dfb7ba88e1fdaeba48 WatchSource:0}: Error finding container 9984707238c3ba711cf926b702d59e1340a6e545bce953dfb7ba88e1fdaeba48: Status 404 returned error can't find the container with id 9984707238c3ba711cf926b702d59e1340a6e545bce953dfb7ba88e1fdaeba48 Oct 29 22:06:09.761977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:09.761928 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:06:10.045463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:10.045372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" event={"ID":"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4","Type":"ContainerStarted","Data":"9984707238c3ba711cf926b702d59e1340a6e545bce953dfb7ba88e1fdaeba48"} Oct 29 22:06:11.055719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:11.055686 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" event={"ID":"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4","Type":"ContainerStarted","Data":"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c"} Oct 29 22:06:11.056162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:11.055814 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:11.104721 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:11.104657 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" podStartSLOduration=1.744977885 podStartE2EDuration="2.104638946s" totalImagesPullingTime="359.661061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:11.101406192 +0000 UTC m=+1429.949883041" watchObservedRunningTime="2026-10-29 22:06:11.104638946 +0000 UTC m=+1429.953115792" Oct 29 22:06:16.434201 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:06:16.434165 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:06:17.068243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.068213 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:06:17.159113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.159081 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:06:17.164326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.164300 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.187269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.187223 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9bb81b-serving-cert\"" Oct 29 22:06:17.192537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.192506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:06:17.243468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.243423 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.243633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.243543 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.243850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.243831 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.243931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.243895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6zs\" (UniqueName: \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.402980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.397338 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.403323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.403293 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6zs\" (UniqueName: \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.403659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.403630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.403984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.403961 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.432047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.413703 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.432269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.427959 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.456933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.443134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.456933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.454069 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6zs\" (UniqueName: \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\") pod \"maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.500258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.500220 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:17.808882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:17.808853 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:06:17.809833 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:06:17.809806 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037b2971_dbc9_4e31_a17e_790c2ad21dab.slice/crio-bbb3d6b7d64144bc431115f5bf90b9c3b755deb40a59eabe05284e93e033ee40 WatchSource:0}: Error finding container bbb3d6b7d64144bc431115f5bf90b9c3b755deb40a59eabe05284e93e033ee40: Status 404 returned error can't find the container with id bbb3d6b7d64144bc431115f5bf90b9c3b755deb40a59eabe05284e93e033ee40 Oct 29 22:06:18.111960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:18.111873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" event={"ID":"037b2971-dbc9-4e31-a17e-790c2ad21dab","Type":"ContainerStarted","Data":"bbb3d6b7d64144bc431115f5bf90b9c3b755deb40a59eabe05284e93e033ee40"} Oct 29 22:06:19.122399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:19.122365 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" event={"ID":"037b2971-dbc9-4e31-a17e-790c2ad21dab","Type":"ContainerStarted","Data":"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df"} Oct 29 22:06:19.122818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:19.122458 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:19.187808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:19.187739 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" podStartSLOduration=1.8395511469999999 podStartE2EDuration="2.187718392s" totalImagesPullingTime="348.167245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:19.170378491 +0000 UTC m=+1438.018855337" watchObservedRunningTime="2026-10-29 22:06:19.187718392 +0000 UTC m=+1438.036195240" Oct 29 22:06:23.346073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:23.346045 4376 scope.go:122] "RemoveContainer" containerID="a422545cf321c68715f0b2519b37323a0b45396e4e27dc21742b16aa57be2a28" Oct 29 22:06:23.356396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:23.356373 4376 scope.go:122] "RemoveContainer" containerID="10b301f17bdc01ee673bc45e74d74c452e2133898d219737a16704f1383debeb" Oct 29 22:06:25.174937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:25.174150 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:06:38.631225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.631192 4376 generic.go:356] "Generic (PLEG): container finished" podID="03f39547-4c22-4c65-8088-7519d945ac7b" containerID="6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2" exitCode=0 Oct 29 22:06:38.631622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.631271 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerDied","Data":"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2"} Oct 29 22:06:38.631745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.631722 4376 scope.go:122] "RemoveContainer" containerID="6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2" Oct 29 22:06:38.633615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.633585 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:38.633691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.633671 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:38.633730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:38.633702 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:39.086078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.086039 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:06:39.086287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.086121 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:06:39.664686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.664597 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerStarted","Data":"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e"} Oct 29 22:06:39.666474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.666449 4376 generic.go:356] "Generic (PLEG): container finished" podID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerID="fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6" exitCode=0 Oct 29 22:06:39.666560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.666503 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerDied","Data":"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6"} Oct 29 22:06:39.666829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.666817 4376 scope.go:122] "RemoveContainer" containerID="fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6" Oct 29 22:06:39.668421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.668388 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:39.668515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.668461 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:39.668515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:39.668494 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:06:40.512352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:40.512323 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:06:40.513338 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:40.513309 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:06:40.513472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:40.513362 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:06:40.692771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:40.692735 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerStarted","Data":"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8"} Oct 29 22:06:40.983002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:40.982859 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:06:41.106030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.105996 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:06:41.106402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.106380 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:41.106686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.106659 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:41.487221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.487125 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:06:41.511937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.511887 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:06:41.512106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:41.511959 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:06:42.096949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:42.096917 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:42.097416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:42.097392 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:42.525927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:42.514703 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:06:42.525927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:42.514762 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:06:43.096859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:43.096827 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:43.097123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:43.097098 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:43.512517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:43.512481 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:06:43.512707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:43.512534 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:06:44.097344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:44.097308 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:44.097744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:44.097356 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:44.516926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:44.516528 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:06:44.516926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:44.516589 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:06:45.096885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:45.096852 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:45.097129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:45.097106 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:45.667407 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:45.667371 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:06:45.678626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:45.678577 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:06:46.099506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:46.099472 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 29 22:06:46.099801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:46.099764 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 29 22:06:46.465452 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:06:46.465411 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:06:46.487707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:46.487664 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" podUID="c2b12b0b-8057-4473-93c4-41349500914b" containerName="payload-processing" containerID="cri-o://4f1767bc48d5f5f5650e2d240d13022dfbf92dc8a947b2390e90435c0b6628ce" gracePeriod=25 Oct 29 22:06:46.841411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:46.841374 4376 generic.go:356] "Generic (PLEG): container finished" podID="c2b12b0b-8057-4473-93c4-41349500914b" containerID="4f1767bc48d5f5f5650e2d240d13022dfbf92dc8a947b2390e90435c0b6628ce" exitCode=0 Oct 29 22:06:46.844008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:46.842244 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" event={"ID":"c2b12b0b-8057-4473-93c4-41349500914b","Type":"ContainerDied","Data":"4f1767bc48d5f5f5650e2d240d13022dfbf92dc8a947b2390e90435c0b6628ce"} Oct 29 22:06:47.177053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.176965 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:06:47.188292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.188262 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\") pod \"c2b12b0b-8057-4473-93c4-41349500914b\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " Oct 29 22:06:47.188456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.188408 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\" (UniqueName: \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\") pod \"c2b12b0b-8057-4473-93c4-41349500914b\" (UID: \"c2b12b0b-8057-4473-93c4-41349500914b\") " Oct 29 22:06:47.188746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.188716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume" pod "c2b12b0b-8057-4473-93c4-41349500914b" (UID: "c2b12b0b-8057-4473-93c4-41349500914b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:47.189044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.189021 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b12b0b-8057-4473-93c4-41349500914b-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:47.190739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.190708 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx" pod "c2b12b0b-8057-4473-93c4-41349500914b" (UID: "c2b12b0b-8057-4473-93c4-41349500914b"). InnerVolumeSpecName "kube-api-access-4rmrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:47.279198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.279163 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:06:47.283317 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.283286 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:06:47.284164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.284144 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:06:47.291348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.291317 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rmrx\" (UniqueName: \"kubernetes.io/projected/c2b12b0b-8057-4473-93c4-41349500914b-kube-api-access-4rmrx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:47.853635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.853603 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" Oct 29 22:06:47.853635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.853600 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" event={"ID":"c2b12b0b-8057-4473-93c4-41349500914b","Type":"ContainerDied","Data":"9fc620b56fa4006bbd7f6372c40d33c6c60b5a07dbf113385aa70acf41871d9a"} Oct 29 22:06:47.854216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.853693 4376 scope.go:122] "RemoveContainer" containerID="4f1767bc48d5f5f5650e2d240d13022dfbf92dc8a947b2390e90435c0b6628ce" Oct 29 22:06:47.915780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.915747 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:06:47.930837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.930808 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6"] Oct 29 22:06:47.956936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:47.956881 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-7046db-67f7fb987b-5gtc6" podUID="c2b12b0b-8057-4473-93c4-41349500914b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded" Oct 29 22:06:47.957068 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:06:47.956946 4376 logging.go:55] [core] [Channel #833 SubChannel #834]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: operation was canceled" Oct 29 22:06:49.273682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:06:49.273649 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b12b0b-8057-4473-93c4-41349500914b" path="/var/lib/kubelet/pods/c2b12b0b-8057-4473-93c4-41349500914b/volumes" Oct 29 22:07:01.309130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.309093 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:07:01.311176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.311150 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c2b12b0b-8057-4473-93c4-41349500914b" containerName="payload-processing" Oct 29 22:07:01.311363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.311334 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b12b0b-8057-4473-93c4-41349500914b" containerName="payload-processing" Oct 29 22:07:01.311777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.311758 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b12b0b-8057-4473-93c4-41349500914b" containerName="payload-processing" Oct 29 22:07:01.317224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.317196 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.327698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.327669 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.328276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.328252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.328401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.328315 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrd79\" (UniqueName: \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.328401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.328363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.335797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.335562 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6153f8-serving-cert\"" Oct 29 22:07:01.336771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.336750 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:07:01.429455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.429419 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.429643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.429506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrd79\" (UniqueName: \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.429643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.429547 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.429732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.429711 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.429897 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:01.429871 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-serving-cert: secret "maas-api-e2e-api-b-6153f8-serving-cert" not found Oct 29 22:07:01.429992 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:01.429978 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls podName:faf65796-bd09-45e0-a101-6e9845d34555 nodeName:}" failed. No retries permitted until 2026-10-29 22:07:01.929956037 +0000 UTC m=+1480.778432882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls") pod "maas-api-e2e-api-b-6153f8-88754995-qlbvv" (UID: "faf65796-bd09-45e0-a101-6e9845d34555") : secret "maas-api-e2e-api-b-6153f8-serving-cert" not found Oct 29 22:07:01.430385 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.430355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.431835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.431818 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.544926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.533233 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrd79\" (UniqueName: \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.955923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.949221 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:01.979931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:01.970744 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") pod \"maas-api-e2e-api-b-6153f8-88754995-qlbvv\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:02.240976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:02.240300 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:02.750014 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:02.746590 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:07:03.106585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:03.106487 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" event={"ID":"faf65796-bd09-45e0-a101-6e9845d34555","Type":"ContainerStarted","Data":"4bf069be821396dcded760c60060284921c3b97d67898aa52b5bc304673cae48"} Oct 29 22:07:04.115267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:04.115177 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" event={"ID":"faf65796-bd09-45e0-a101-6e9845d34555","Type":"ContainerStarted","Data":"3599daa2fbb6907401a3f31e3e7a0f4c87c0dddd64adef7e0c59433a35613d53"} Oct 29 22:07:04.115739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:04.115319 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:04.192670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:04.192602 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" podStartSLOduration=2.487438082 podStartE2EDuration="3.192581656s" totalImagesPullingTime="705.143574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:04.191429156 +0000 UTC m=+1483.039905999" watchObservedRunningTime="2026-10-29 22:07:04.192581656 +0000 UTC m=+1483.041058502" Oct 29 22:07:05.383833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:05.383801 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:07:05.606475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:05.606442 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:07:08.242412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:08.242205 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:07:08.243783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:08.243616 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerName="maas-api" containerID="cri-o://066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f" gracePeriod=30 Oct 29 22:07:08.981927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:08.971808 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.121:8443/health\": dial tcp 10.134.0.121:8443: connect: connection refused" Oct 29 22:07:09.025083 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.023309 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:07:09.205973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.201067 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:07:09.403250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.403214 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:07:09.422182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.422141 4376 generic.go:356] "Generic (PLEG): container finished" podID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerID="066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f" exitCode=0 Oct 29 22:07:09.423100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.422614 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" Oct 29 22:07:09.423600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.422682 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" event={"ID":"0e1e669b-d650-4cfa-8500-7cfa709ee4a4","Type":"ContainerDied","Data":"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f"} Oct 29 22:07:09.423822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.423793 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88" event={"ID":"0e1e669b-d650-4cfa-8500-7cfa709ee4a4","Type":"ContainerDied","Data":"71f36633132c3b9242f6b2d9a813f97d8ee11b9f393808510b85a175798f79d8"} Oct 29 22:07:09.424099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.424053 4376 scope.go:122] "RemoveContainer" containerID="066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f" Oct 29 22:07:09.477942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.476080 4376 scope.go:122] "RemoveContainer" containerID="066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f" Oct 29 22:07:09.480472 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:09.480359 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f\": container with ID starting with 066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f not found: ID does not exist" containerID="066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f" Oct 29 22:07:09.480472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.480412 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f"} err="failed to get container status \"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f\": rpc error: code = NotFound desc = could not find container \"066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f\": container with ID starting with 066aa5b981a70e4a68d163f726f4d52ec038eda24c62d75c54e17d74d8afe60f not found: ID does not exist" Oct 29 22:07:09.579025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.578877 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\") pod \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " Oct 29 22:07:09.579186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.579077 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\") pod \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " Oct 29 22:07:09.579186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.579165 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") pod \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " Oct 29 22:07:09.579329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.579300 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\" (UniqueName: \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\") pod \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\" (UID: \"0e1e669b-d650-4cfa-8500-7cfa709ee4a4\") " Oct 29 22:07:09.579527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.579489 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca" pod "0e1e669b-d650-4cfa-8500-7cfa709ee4a4" (UID: "0e1e669b-d650-4cfa-8500-7cfa709ee4a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:09.580095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.580069 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.582523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.581572 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs" pod "0e1e669b-d650-4cfa-8500-7cfa709ee4a4" (UID: "0e1e669b-d650-4cfa-8500-7cfa709ee4a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:09.582523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.581800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls" pod "0e1e669b-d650-4cfa-8500-7cfa709ee4a4" (UID: "0e1e669b-d650-4cfa-8500-7cfa709ee4a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:09.582523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.581843 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf" pod "0e1e669b-d650-4cfa-8500-7cfa709ee4a4" (UID: "0e1e669b-d650-4cfa-8500-7cfa709ee4a4"). InnerVolumeSpecName "kube-api-access-v4vjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:09.681758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.681714 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.681758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.681753 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.681758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.681762 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vjf\" (UniqueName: \"kubernetes.io/projected/0e1e669b-d650-4cfa-8500-7cfa709ee4a4-kube-api-access-v4vjf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.856016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.845164 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:07:09.856016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:09.850268 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7046db-568dc7c7cb-6gs88"] Oct 29 22:07:10.130185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:10.130097 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:07:11.278395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:11.278345 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" path="/var/lib/kubelet/pods/0e1e669b-d650-4cfa-8500-7cfa709ee4a4/volumes" Oct 29 22:07:15.481115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.481080 4376 generic.go:356] "Generic (PLEG): container finished" podID="789c0500-3aab-4019-bc74-835b516fd5eb" containerID="cfc04ac62d80faa7c6fb85d234916ab259f5eed19a6412ce56214bbcf031f4a7" exitCode=0 Oct 29 22:07:15.481601 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.481160 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerDied","Data":"cfc04ac62d80faa7c6fb85d234916ab259f5eed19a6412ce56214bbcf031f4a7"} Oct 29 22:07:15.481758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.481736 4376 scope.go:122] "RemoveContainer" containerID="cfc04ac62d80faa7c6fb85d234916ab259f5eed19a6412ce56214bbcf031f4a7" Oct 29 22:07:15.483143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483122 4376 generic.go:356] "Generic (PLEG): container finished" podID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerID="b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc" exitCode=0 Oct 29 22:07:15.483249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerDied","Data":"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc"} Oct 29 22:07:15.483637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483619 4376 scope.go:122] "RemoveContainer" containerID="b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc" Oct 29 22:07:15.483810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483780 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:15.483875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483861 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:15.483943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.483922 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:15.485464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.485433 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:15.485559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.485503 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:15.485559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:15.485537 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:07:16.436434 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:16.436395 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:07:16.503481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.503440 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerStarted","Data":"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636"} Oct 29 22:07:16.505550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.505524 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerStarted","Data":"9f15baf346e5765a66a1065ca9223a52807d2d9e36ea3f76ac0bd33615fa5b27"} Oct 29 22:07:16.886391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.882306 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:07:16.886391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.882555 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:16.886391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.882819 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:16.902085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:16.902040 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:07:17.424321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:17.424279 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:07:17.425392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:17.425369 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:17.425487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:17.425429 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:17.882412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:17.881837 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:17.883116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:17.883056 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:18.424225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:18.424189 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:18.424422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:18.424245 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:18.886858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:18.882527 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:18.886858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:18.882600 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:19.486961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:19.460631 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:19.486961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:19.460709 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:19.886992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:19.879694 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:19.886992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:19.879790 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:20.431624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:20.431585 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:20.431957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:20.431919 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:20.891948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:20.885644 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:20.891948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:20.885734 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:21.418898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.418828 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:07:21.426265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.426226 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:21.426894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.426549 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:21.724048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.724012 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:07:21.878927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.878559 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:21.878927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:21.878635 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:22.424138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:22.424109 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:22.424603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:22.424579 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:22.880019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:22.879982 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 29 22:07:22.880314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:22.880288 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 29 22:07:23.424247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:23.424207 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:07:23.424655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:23.424285 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 29 22:07:23.884155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:23.884122 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:07:23.885418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:23.885396 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:07:24.429958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:24.429927 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:07:24.430974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:24.430952 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:07:31.274929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.274878 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:07:31.275701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.275666 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" podUID="1994c957-9487-4b4c-a890-1400fc54b265" containerName="payload-processing" containerID="cri-o://31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14" gracePeriod=30 Oct 29 22:07:31.286645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.286612 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:07:31.286929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.286877 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" podUID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" containerName="payload-pre-processing" containerID="cri-o://05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a" gracePeriod=30 Oct 29 22:07:31.597565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.597536 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:07:31.630926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.630881 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:07:31.711960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.698540 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\" (UniqueName: \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\") pod \"1994c957-9487-4b4c-a890-1400fc54b265\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " Oct 29 22:07:31.711960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.698698 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\" (UniqueName: \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\") pod \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " Oct 29 22:07:31.723190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.722234 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\") pod \"1994c957-9487-4b4c-a890-1400fc54b265\" (UID: \"1994c957-9487-4b4c-a890-1400fc54b265\") " Oct 29 22:07:31.723190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.722290 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\") pod \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\" (UID: \"89d83b5c-eace-4a54-b4b1-39cb143c35ad\") " Oct 29 22:07:31.753065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.751320 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume" pod "1994c957-9487-4b4c-a890-1400fc54b265" (UID: "1994c957-9487-4b4c-a890-1400fc54b265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:31.783927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.770167 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume" pod "89d83b5c-eace-4a54-b4b1-39cb143c35ad" (UID: "89d83b5c-eace-4a54-b4b1-39cb143c35ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:31.821602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.821493 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf" pod "1994c957-9487-4b4c-a890-1400fc54b265" (UID: "1994c957-9487-4b4c-a890-1400fc54b265"). InnerVolumeSpecName "kube-api-access-8r7jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:31.918926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.875153 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x" pod "89d83b5c-eace-4a54-b4b1-39cb143c35ad" (UID: "89d83b5c-eace-4a54-b4b1-39cb143c35ad"). InnerVolumeSpecName "kube-api-access-p6v5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:31.945933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.924578 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d83b5c-eace-4a54-b4b1-39cb143c35ad-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.945933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.924623 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7jf\" (UniqueName: \"kubernetes.io/projected/1994c957-9487-4b4c-a890-1400fc54b265-kube-api-access-8r7jf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.945933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.924637 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6v5x\" (UniqueName: \"kubernetes.io/projected/89d83b5c-eace-4a54-b4b1-39cb143c35ad-kube-api-access-p6v5x\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.945933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:31.924651 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994c957-9487-4b4c-a890-1400fc54b265-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:32.327611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.327568 4376 generic.go:356] "Generic (PLEG): container finished" podID="1994c957-9487-4b4c-a890-1400fc54b265" containerID="31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14" exitCode=0 Oct 29 22:07:32.328125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.327618 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" Oct 29 22:07:32.328125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.327652 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" event={"ID":"1994c957-9487-4b4c-a890-1400fc54b265","Type":"ContainerDied","Data":"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14"} Oct 29 22:07:32.328125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.327723 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx" event={"ID":"1994c957-9487-4b4c-a890-1400fc54b265","Type":"ContainerDied","Data":"89e70e5eac5273bbf450ebcd28455c8bef507608438ac13bca9374ec7378ba6d"} Oct 29 22:07:32.328125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.327757 4376 scope.go:122] "RemoveContainer" containerID="31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14" Oct 29 22:07:32.329359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.329334 4376 generic.go:356] "Generic (PLEG): container finished" podID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" containerID="05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a" exitCode=0 Oct 29 22:07:32.329451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.329388 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" event={"ID":"89d83b5c-eace-4a54-b4b1-39cb143c35ad","Type":"ContainerDied","Data":"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a"} Oct 29 22:07:32.329451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.329390 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" Oct 29 22:07:32.329451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.329428 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq" event={"ID":"89d83b5c-eace-4a54-b4b1-39cb143c35ad","Type":"ContainerDied","Data":"26aba0b0fda782b7c2ef25f75c253cab317add9877b025e6a83d2d6c0bb3fa66"} Oct 29 22:07:32.339071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.339049 4376 scope.go:122] "RemoveContainer" containerID="31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14" Oct 29 22:07:32.339402 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:32.339384 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14\": container with ID starting with 31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14 not found: ID does not exist" containerID="31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14" Oct 29 22:07:32.339457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.339411 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14"} err="failed to get container status \"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14\": rpc error: code = NotFound desc = could not find container \"31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14\": container with ID starting with 31d841caa305d0445a9df89fe05e806204ddfa14165620edff267055a5980d14 not found: ID does not exist" Oct 29 22:07:32.339457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.339433 4376 scope.go:122] "RemoveContainer" containerID="05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a" Oct 29 22:07:32.350610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.350584 4376 scope.go:122] "RemoveContainer" containerID="05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a" Oct 29 22:07:32.350986 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:32.350960 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a\": container with ID starting with 05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a not found: ID does not exist" containerID="05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a" Oct 29 22:07:32.351075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.351000 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a"} err="failed to get container status \"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a\": rpc error: code = NotFound desc = could not find container \"05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a\": container with ID starting with 05d8c9a97cd12944ff22694bc4e35085cbfdc38670a2f9b6e655247520ed196a not found: ID does not exist" Oct 29 22:07:32.392594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.392550 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:07:32.408408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.408365 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-587bcd69b8-cmvhx"] Oct 29 22:07:32.441529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.441486 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:07:32.446914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:32.446869 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-75ddf84fcf-c25dq"] Oct 29 22:07:33.280375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:33.280339 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1994c957-9487-4b4c-a890-1400fc54b265" path="/var/lib/kubelet/pods/1994c957-9487-4b4c-a890-1400fc54b265/volumes" Oct 29 22:07:33.281172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:33.281149 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" path="/var/lib/kubelet/pods/89d83b5c-eace-4a54-b4b1-39cb143c35ad/volumes" Oct 29 22:07:37.501711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.501666 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:07:37.502918 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502859 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerName="maas-api" Oct 29 22:07:37.502918 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502890 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerName="maas-api" Oct 29 22:07:37.503177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502930 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1994c957-9487-4b4c-a890-1400fc54b265" containerName="payload-processing" Oct 29 22:07:37.503177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502939 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1994c957-9487-4b4c-a890-1400fc54b265" containerName="payload-processing" Oct 29 22:07:37.503177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502952 4376 cpu_manager.go:414] "containerMap: removing container" podUID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" containerName="payload-pre-processing" Oct 29 22:07:37.503177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.502960 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" containerName="payload-pre-processing" Oct 29 22:07:37.503441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.503189 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1e669b-d650-4cfa-8500-7cfa709ee4a4" containerName="maas-api" Oct 29 22:07:37.503441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.503206 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1994c957-9487-4b4c-a890-1400fc54b265" containerName="payload-processing" Oct 29 22:07:37.503441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.503219 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d83b5c-eace-4a54-b4b1-39cb143c35ad" containerName="payload-pre-processing" Oct 29 22:07:37.508119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.508091 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.527033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.526995 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:07:37.532923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.532880 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3fe45b\"/\"default-dockercfg-h9mjz\"" Oct 29 22:07:37.533076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.532956 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3fe45b\"/\"ipp-route-3fe45b-kserve-self-signed-certs\"" Oct 29 22:07:37.539390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.539347 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3fe45b\"/\"openshift-service-ca.crt\"" Oct 29 22:07:37.539638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.539615 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3fe45b\"/\"kube-root-ca.crt\"" Oct 29 22:07:37.597486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.597450 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.597683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.597493 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.597787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.597765 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.597915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.597888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.597977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.597963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmngk\" (UniqueName: \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.598057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.598044 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmngk\" (UniqueName: \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699294 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699314 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699340 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699574 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699642 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699806 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699695 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.699980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.699955 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.701627 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.701599 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.702152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.702131 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.750929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.750875 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmngk\" (UniqueName: \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\") pod \"ipp-route-3fe45b-kserve-587d45885-wfwjw\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:37.825345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:37.825228 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:38.481458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:38.479067 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:07:39.485347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:39.485311 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" event={"ID":"acd40135-543f-4b81-9d20-964f0c60a12f","Type":"ContainerStarted","Data":"0fb4fc6aa6d222ddc1b811f8a7c51970ddf6000859a37f26ec21ca668b41a633"} Oct 29 22:07:39.485347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:39.485354 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" event={"ID":"acd40135-543f-4b81-9d20-964f0c60a12f","Type":"ContainerStarted","Data":"df0730f44e7ed7b1b8a0103c2e841b111b0b39548c1bc8c3cfa1c540bd81c4d2"} Oct 29 22:07:39.532788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:39.532733 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podStartSLOduration=2.532715904 podStartE2EDuration="2.532715904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:39.530664525 +0000 UTC m=+1518.379141594" watchObservedRunningTime="2026-10-29 22:07:39.532715904 +0000 UTC m=+1518.381192749" Oct 29 22:07:39.826033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:39.825953 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:40.241097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.241064 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:07:40.244790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.244770 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.268595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.268554 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:07:40.277015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.276984 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9d9498-serving-cert\"" Oct 29 22:07:40.325074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.325035 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.325286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.325092 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.325853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.325826 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.325997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.325891 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2bh\" (UniqueName: \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.428659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.428618 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.447939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.429403 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.448366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.448330 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.459643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.449411 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2bh\" (UniqueName: \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.460499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.449308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.480020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.437838 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.480281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.463619 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.520984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.507599 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2bh\" (UniqueName: \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\") pod \"maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:40.560853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:40.560575 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:41.084931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:41.070353 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:07:41.561924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:41.548860 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" event={"ID":"63b6cccb-2039-46d1-8843-0cd1fd5ce95f","Type":"ContainerStarted","Data":"6f50ff9efb32e0cc795a9bd822764fd08850b702c71b2db00bbf88beb87f8eb5"} Oct 29 22:07:42.556752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:42.556713 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" event={"ID":"63b6cccb-2039-46d1-8843-0cd1fd5ce95f","Type":"ContainerStarted","Data":"b261100780904d3868b83c1352a4c4ff44af8c7724bcf209c5d3bec6737310e6"} Oct 29 22:07:42.556974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:42.556790 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:42.635934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:42.635281 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" podStartSLOduration=2.101502648 podStartE2EDuration="2.622059164s" totalImagesPullingTime="520.556516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:42.61849519 +0000 UTC m=+1521.466972041" watchObservedRunningTime="2026-10-29 22:07:42.622059164 +0000 UTC m=+1521.470536038" Oct 29 22:07:46.511813 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:46.511763 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:07:47.825870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:47.825829 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:47.843869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:47.843842 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:48.571483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:48.571452 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:07:48.633368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:48.633330 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:07:54.709258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:54.709217 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 22:07:54.709810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:54.709454 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" containerID="cri-o://1fb6b5a5eef6dae2f278eaf765e80aa9193abeeea08a2c3a41dac235c2bd862e" gracePeriod=30 Oct 29 22:07:54.722810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:54.722776 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 22:07:54.723093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:54.723070 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" containerID="cri-o://4f2d1d058323b4c2dc6100a5dfd82621c3a4c3c3a603d7d7c03553003829893c" gracePeriod=30 Oct 29 22:07:56.882334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:56.882299 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:07:56.883408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:56.883372 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" podUID="1adb92bb-36c8-41b8-a07e-d2995bda710b" containerName="maas-api" containerID="cri-o://3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c" gracePeriod=30 Oct 29 22:07:57.223354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.223319 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:07:57.353269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.353238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\") pod \"1adb92bb-36c8-41b8-a07e-d2995bda710b\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " Oct 29 22:07:57.353480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.353288 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\") pod \"1adb92bb-36c8-41b8-a07e-d2995bda710b\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " Oct 29 22:07:57.353480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.353354 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\") pod \"1adb92bb-36c8-41b8-a07e-d2995bda710b\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " Oct 29 22:07:57.353480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.353447 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\" (UniqueName: \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\") pod \"1adb92bb-36c8-41b8-a07e-d2995bda710b\" (UID: \"1adb92bb-36c8-41b8-a07e-d2995bda710b\") " Oct 29 22:07:57.353784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.353756 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca" pod "1adb92bb-36c8-41b8-a07e-d2995bda710b" (UID: "1adb92bb-36c8-41b8-a07e-d2995bda710b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:57.354926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.354881 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adb92bb-36c8-41b8-a07e-d2995bda710b-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.355944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.355899 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs" pod "1adb92bb-36c8-41b8-a07e-d2995bda710b" (UID: "1adb92bb-36c8-41b8-a07e-d2995bda710b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:57.356074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.356049 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls" pod "1adb92bb-36c8-41b8-a07e-d2995bda710b" (UID: "1adb92bb-36c8-41b8-a07e-d2995bda710b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:57.356144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.356075 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6" pod "1adb92bb-36c8-41b8-a07e-d2995bda710b" (UID: "1adb92bb-36c8-41b8-a07e-d2995bda710b"). InnerVolumeSpecName "kube-api-access-rm8l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:57.475956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.460718 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm8l6\" (UniqueName: \"kubernetes.io/projected/1adb92bb-36c8-41b8-a07e-d2995bda710b-kube-api-access-rm8l6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.475956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.460762 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.475956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.460777 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1adb92bb-36c8-41b8-a07e-d2995bda710b-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.777613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.777526 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" Oct 29 22:07:57.777789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.777604 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" event={"ID":"1adb92bb-36c8-41b8-a07e-d2995bda710b","Type":"ContainerDied","Data":"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c"} Oct 29 22:07:57.777789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.777702 4376 scope.go:122] "RemoveContainer" containerID="3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c" Oct 29 22:07:57.779309 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.779157 4376 generic.go:356] "Generic (PLEG): container finished" podID="1adb92bb-36c8-41b8-a07e-d2995bda710b" containerID="3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c" exitCode=0 Oct 29 22:07:57.779393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.779370 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g" event={"ID":"1adb92bb-36c8-41b8-a07e-d2995bda710b","Type":"ContainerDied","Data":"f9f42059cc89a8f3299bce006a5a9f4a767a028b6cd9da17d3ff11c2967ca7f8"} Oct 29 22:07:57.825935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.809182 4376 scope.go:122] "RemoveContainer" containerID="3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c" Oct 29 22:07:57.825935 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:57.812183 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c\": container with ID starting with 3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c not found: ID does not exist" containerID="3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c" Oct 29 22:07:57.825935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.812233 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c"} err="failed to get container status \"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c\": rpc error: code = NotFound desc = could not find container \"3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c\": container with ID starting with 3ef32d72ef5e3a1712e8dce9f6d5b69d64e8aab5a310e27316a4aa66e1720d2c not found: ID does not exist" Oct 29 22:07:57.943687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.943647 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:07:57.948524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:57.948499 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b16b28-6fbd6876c4-dss2g"] Oct 29 22:07:59.274332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.274296 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adb92bb-36c8-41b8-a07e-d2995bda710b" path="/var/lib/kubelet/pods/1adb92bb-36c8-41b8-a07e-d2995bda710b/volumes" Oct 29 22:07:59.449559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.449519 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:07:59.450318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.450302 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1adb92bb-36c8-41b8-a07e-d2995bda710b" containerName="maas-api" Oct 29 22:07:59.450363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.450320 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adb92bb-36c8-41b8-a07e-d2995bda710b" containerName="maas-api" Oct 29 22:07:59.450478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.450468 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adb92bb-36c8-41b8-a07e-d2995bda710b" containerName="maas-api" Oct 29 22:07:59.453731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.453712 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.485173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.485146 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9b0939-serving-cert\"" Oct 29 22:07:59.490005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.489972 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:07:59.526741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.526653 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.526932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.526806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.527149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.527123 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.527284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.527251 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ss6\" (UniqueName: \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.628868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.628827 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.629110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.629043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.629181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.629124 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ss6\" (UniqueName: \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.629237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.629222 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.629297 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:59.629228 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-serving-cert: secret "maas-api-e2e-praxis-b-9b0939-serving-cert" not found Oct 29 22:07:59.629353 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:07:59.629308 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls podName:70d79280-22ec-4971-a74a-96410efaeb96 nodeName:}" failed. No retries permitted until 2026-10-29 22:08:00.129285904 +0000 UTC m=+1538.977762751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls") pod "maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" (UID: "70d79280-22ec-4971-a74a-96410efaeb96") : secret "maas-api-e2e-praxis-b-9b0939-serving-cert" not found Oct 29 22:07:59.629742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.629709 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.631775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.631752 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.755927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.752986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ss6\" (UniqueName: \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:07:59.901150 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.900812 4376 generic.go:356] "Generic (PLEG): container finished" podID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerID="4f2d1d058323b4c2dc6100a5dfd82621c3a4c3c3a603d7d7c03553003829893c" exitCode=0 Oct 29 22:07:59.901150 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.901057 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" event={"ID":"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21","Type":"ContainerDied","Data":"4f2d1d058323b4c2dc6100a5dfd82621c3a4c3c3a603d7d7c03553003829893c"} Oct 29 22:07:59.906193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.906149 4376 generic.go:356] "Generic (PLEG): container finished" podID="95a26720-f408-4ad2-a6a6-01317092e316" containerID="1fb6b5a5eef6dae2f278eaf765e80aa9193abeeea08a2c3a41dac235c2bd862e" exitCode=0 Oct 29 22:07:59.906360 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:07:59.906235 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" event={"ID":"95a26720-f408-4ad2-a6a6-01317092e316","Type":"ContainerDied","Data":"1fb6b5a5eef6dae2f278eaf765e80aa9193abeeea08a2c3a41dac235c2bd862e"} Oct 29 22:08:00.065459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.065421 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 22:08:00.089881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.089817 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 22:08:00.163285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163249 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.163465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163338 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.163465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163413 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.163465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163461 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163519 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163552 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163707 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163760 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.163942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.163791 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.164299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.164065 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:00.164299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.164086 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:00.164601 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.164572 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.165177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.165097 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.165434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.165361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.167875 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168010 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168105 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168222 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\") pod \"95a26720-f408-4ad2-a6a6-01317092e316\" (UID: \"95a26720-f408-4ad2-a6a6-01317092e316\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168283 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168317 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168359 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168423 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.168530 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\") pod \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\" (UID: \"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21\") " Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169305 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169430 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169480 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "kube-api-access-hgprv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169683 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169704 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169720 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169733 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169746 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a26720-f408-4ad2-a6a6-01317092e316-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169759 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169771 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169784 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgprv\" (UniqueName: \"kubernetes.io/projected/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-kube-api-access-hgprv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169695 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.169869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.170190 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.170613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.170423 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.173504 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.173584 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.174151 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo" pod "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" (UID: "1f9e9abf-5904-4bb5-ac7c-6fff634c0a21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.174222 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.174658 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.175267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:08:00.179430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.178625 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz" pod "95a26720-f408-4ad2-a6a6-01317092e316" (UID: "95a26720-f408-4ad2-a6a6-01317092e316"). InnerVolumeSpecName "kube-api-access-tp2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:00.271063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271025 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271059 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271068 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a26720-f408-4ad2-a6a6-01317092e316-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271078 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2fz\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-kube-api-access-tp2fz\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271087 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a26720-f408-4ad2-a6a6-01317092e316-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271096 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a26720-f408-4ad2-a6a6-01317092e316-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271105 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271112 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271119 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.271315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.271127 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.368444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.368348 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:08:00.776926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.775862 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:08:00.921505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.921413 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" Oct 29 22:08:00.921505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.921444 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99" event={"ID":"95a26720-f408-4ad2-a6a6-01317092e316","Type":"ContainerDied","Data":"0c372ebe6268d51edc18c6c50471fa608087530d34b8606bc1198158d6e009a0"} Oct 29 22:08:00.921769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.921519 4376 scope.go:122] "RemoveContainer" containerID="1fb6b5a5eef6dae2f278eaf765e80aa9193abeeea08a2c3a41dac235c2bd862e" Oct 29 22:08:00.923293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.923264 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" event={"ID":"70d79280-22ec-4971-a74a-96410efaeb96","Type":"ContainerStarted","Data":"5ee9e9427e9b0cdc3a1d6484dc1c94d04f6f26d445a474a570f7e5bb305fbc88"} Oct 29 22:08:00.925135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.925106 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" event={"ID":"1f9e9abf-5904-4bb5-ac7c-6fff634c0a21","Type":"ContainerDied","Data":"63d424bf29f7512914cc57c2920ce2c82c27690717ba219f74b1b378efb762df"} Oct 29 22:08:00.925262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.925244 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr" Oct 29 22:08:00.934929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.934756 4376 scope.go:122] "RemoveContainer" containerID="4f2d1d058323b4c2dc6100a5dfd82621c3a4c3c3a603d7d7c03553003829893c" Oct 29 22:08:00.985419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.985372 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 22:08:00.999003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:00.998966 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j5t99"] Oct 29 22:08:01.048868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.048825 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 22:08:01.054325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.054288 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-2ccqr"] Oct 29 22:08:01.278246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.278212 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" path="/var/lib/kubelet/pods/1f9e9abf-5904-4bb5-ac7c-6fff634c0a21/volumes" Oct 29 22:08:01.278800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.278784 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a26720-f408-4ad2-a6a6-01317092e316" path="/var/lib/kubelet/pods/95a26720-f408-4ad2-a6a6-01317092e316/volumes" Oct 29 22:08:01.936063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.936026 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" event={"ID":"70d79280-22ec-4971-a74a-96410efaeb96","Type":"ContainerStarted","Data":"074fb626f91153e5e16718b84b47915bcb665fa55b206cd8b9a44c589e83f5a2"} Oct 29 22:08:01.936602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.936092 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:08:01.983890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:01.983835 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" podStartSLOduration=2.653490439 podStartE2EDuration="2.983819554s" totalImagesPullingTime="330.329115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:01.98153305 +0000 UTC m=+1540.830009896" watchObservedRunningTime="2026-10-29 22:08:01.983819554 +0000 UTC m=+1540.832296401" Oct 29 22:08:05.912353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:05.912321 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:08:05.912870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:05.912544 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" containerID="cri-o://95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e" gracePeriod=30 Oct 29 22:08:07.951717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:07.951687 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:08:11.209844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.209807 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:08:11.211183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.211150 4376 generic.go:356] "Generic (PLEG): container finished" podID="03f39547-4c22-4c65-8088-7519d945ac7b" containerID="95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e" exitCode=0 Oct 29 22:08:11.211274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.211217 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerDied","Data":"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e"} Oct 29 22:08:11.211314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.211275 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" event={"ID":"03f39547-4c22-4c65-8088-7519d945ac7b","Type":"ContainerDied","Data":"fee20615358781c96a8ce0e2d4c8bbada20705cbfe7d93ce27efddcb8647d9ed"} Oct 29 22:08:11.211314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.211299 4376 scope.go:122] "RemoveContainer" containerID="95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e" Oct 29 22:08:11.223088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.223055 4376 scope.go:122] "RemoveContainer" containerID="6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2" Oct 29 22:08:11.234688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.234664 4376 scope.go:122] "RemoveContainer" containerID="95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e" Oct 29 22:08:11.235090 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:08:11.235067 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e\": container with ID starting with 95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e not found: ID does not exist" containerID="95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e" Oct 29 22:08:11.235183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.235104 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e"} err="failed to get container status \"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e\": rpc error: code = NotFound desc = could not find container \"95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e\": container with ID starting with 95561794db57d0b641525b88084900a77b7cf0b51f3f17906998e7bf2bbccd7e not found: ID does not exist" Oct 29 22:08:11.235183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.235123 4376 scope.go:122] "RemoveContainer" containerID="6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2" Oct 29 22:08:11.235457 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:08:11.235437 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2\": container with ID starting with 6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2 not found: ID does not exist" containerID="6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2" Oct 29 22:08:11.235492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.235468 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2"} err="failed to get container status \"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2\": rpc error: code = NotFound desc = could not find container \"6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2\": container with ID starting with 6d0597ba61e77f4a79fbe7f082f89a591f897e30b588732387a87de6e14f7fa2 not found: ID does not exist" Oct 29 22:08:11.266566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.266524 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.266588 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.266770 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.266815 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267027 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267085 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267130 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267253 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267277 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:11.267479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267307 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\") pod \"03f39547-4c22-4c65-8088-7519d945ac7b\" (UID: \"03f39547-4c22-4c65-8088-7519d945ac7b\") " Oct 29 22:08:11.267592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267508 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:11.267999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.267970 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:11.268152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.268104 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:11.268306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.268284 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:11.268386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.268289 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.268386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.268331 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.268386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.268348 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.269380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.269347 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:11.269471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.269399 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "kube-api-access-hm9dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:11.269526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.269464 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:11.269874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.269852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo" pod "03f39547-4c22-4c65-8088-7519d945ac7b" (UID: "03f39547-4c22-4c65-8088-7519d945ac7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:08:11.370763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370732 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.370763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370760 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f39547-4c22-4c65-8088-7519d945ac7b-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.370763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370769 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f39547-4c22-4c65-8088-7519d945ac7b-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.371039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370791 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.371039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370810 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm9dj\" (UniqueName: \"kubernetes.io/projected/03f39547-4c22-4c65-8088-7519d945ac7b-kube-api-access-hm9dj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:11.371039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:11.370820 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f39547-4c22-4c65-8088-7519d945ac7b-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:12.219845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:12.219813 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq" Oct 29 22:08:12.295682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:12.295639 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:08:12.308964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:12.308899 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b16b28-openshift-default-78b47d7f79-6wksq"] Oct 29 22:08:13.284375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:13.284327 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" path="/var/lib/kubelet/pods/03f39547-4c22-4c65-8088-7519d945ac7b/volumes" Oct 29 22:08:16.438607 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:08:16.438565 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:08:26.643659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.643618 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:08:26.644835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644801 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" Oct 29 22:08:26.644835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644833 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" Oct 29 22:08:26.645047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644869 4376 cpu_manager.go:414] "containerMap: removing container" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:26.645047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644878 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:26.645047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644965 4376 cpu_manager.go:414] "containerMap: removing container" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" Oct 29 22:08:26.645047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.644974 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" Oct 29 22:08:26.645257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.645154 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9e9abf-5904-4bb5-ac7c-6fff634c0a21" containerName="istio-proxy" Oct 29 22:08:26.645257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.645177 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:26.645257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.645193 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:26.645257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.645207 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a26720-f408-4ad2-a6a6-01317092e316" containerName="istio-proxy" Oct 29 22:08:26.651763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.651717 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.672972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.672940 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:08:26.823063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823146 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823183 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rs5\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823244 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823329 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823480 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823530 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823639 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.823700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.823686 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925077 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925153 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925224 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925255 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rs5\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925281 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925361 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925559 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925715 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925743 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.925893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925868 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.926083 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.925983 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.926184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.926160 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.926248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.926174 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.927738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.927708 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.928123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.928105 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.951724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.951691 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.951949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.951797 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rs5\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:26.974455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:26.974417 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:27.299750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.299717 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:08:27.300731 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:08:27.300704 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d7191d_4570_4090_8589_8e73dd50f7c6.slice/crio-bd2a1ddda8c00c97da2f94b5cd88451ce97f6afc7623cb82a30d8fc942ce1fc0 WatchSource:0}: Error finding container bd2a1ddda8c00c97da2f94b5cd88451ce97f6afc7623cb82a30d8fc942ce1fc0: Status 404 returned error can't find the container with id bd2a1ddda8c00c97da2f94b5cd88451ce97f6afc7623cb82a30d8fc942ce1fc0 Oct 29 22:08:27.303644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.303608 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:08:27.303704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.303683 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:08:27.303743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.303711 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:08:27.487200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.487150 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" event={"ID":"19d7191d-4570-4090-8589-8e73dd50f7c6","Type":"ContainerStarted","Data":"84e62862974aa5b4c09906a81496833c2dabba7441e702c19bde2c1ae59144f8"} Oct 29 22:08:27.487413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.487272 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" event={"ID":"19d7191d-4570-4090-8589-8e73dd50f7c6","Type":"ContainerStarted","Data":"bd2a1ddda8c00c97da2f94b5cd88451ce97f6afc7623cb82a30d8fc942ce1fc0"} Oct 29 22:08:27.538525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.538453 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podStartSLOduration=1.538437652 podStartE2EDuration="1.538437652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:27.538073854 +0000 UTC m=+1566.386550700" watchObservedRunningTime="2026-10-29 22:08:27.538437652 +0000 UTC m=+1566.386914497" Oct 29 22:08:27.975412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:27.975374 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:28.976831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:28.976792 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:28.977282 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:28.976875 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:29.332033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:29.331946 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:08:29.364607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:29.364572 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:08:29.975525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:29.975485 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:29.975711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:29.975552 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:30.975493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:30.975453 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:30.975887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:30.975513 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:31.975242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:31.975196 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:31.975462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:31.975271 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:32.980211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:32.980172 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:32.981244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:32.980793 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:33.986482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:33.986381 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:33.987301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:33.987259 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:34.332434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.332322 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" podUID="96a035a3-d117-4936-9fb0-9f3ed354d32a" containerName="payload-processing" containerID="cri-o://7293e31127a9490d0c2c702a4eba744eda523cefca7eed82cc596686fd62eb5d" gracePeriod=25 Oct 29 22:08:34.366098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.365021 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" podUID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" containerName="payload-pre-processing" containerID="cri-o://125c7d9a362ee5c0a79584f06261efb7fbdf0e0a30665b194d94ccec04709cf4" gracePeriod=25 Oct 29 22:08:34.777082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.777042 4376 generic.go:356] "Generic (PLEG): container finished" podID="96a035a3-d117-4936-9fb0-9f3ed354d32a" containerID="7293e31127a9490d0c2c702a4eba744eda523cefca7eed82cc596686fd62eb5d" exitCode=0 Oct 29 22:08:34.777517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.777488 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" event={"ID":"96a035a3-d117-4936-9fb0-9f3ed354d32a","Type":"ContainerDied","Data":"7293e31127a9490d0c2c702a4eba744eda523cefca7eed82cc596686fd62eb5d"} Oct 29 22:08:34.812009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.811696 4376 generic.go:356] "Generic (PLEG): container finished" podID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" containerID="125c7d9a362ee5c0a79584f06261efb7fbdf0e0a30665b194d94ccec04709cf4" exitCode=0 Oct 29 22:08:34.812009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.811850 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" event={"ID":"88eca3d2-8cfb-413a-bf90-d0b554072dc3","Type":"ContainerDied","Data":"125c7d9a362ee5c0a79584f06261efb7fbdf0e0a30665b194d94ccec04709cf4"} Oct 29 22:08:34.992999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.992308 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:34.992999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:34.992370 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:35.518194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.518110 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:08:35.521639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.521611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\") pod \"96a035a3-d117-4936-9fb0-9f3ed354d32a\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " Oct 29 22:08:35.522518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.521914 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\" (UniqueName: \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\") pod \"96a035a3-d117-4936-9fb0-9f3ed354d32a\" (UID: \"96a035a3-d117-4936-9fb0-9f3ed354d32a\") " Oct 29 22:08:35.522518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.522173 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume" pod "96a035a3-d117-4936-9fb0-9f3ed354d32a" (UID: "96a035a3-d117-4936-9fb0-9f3ed354d32a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:35.522518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.522493 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a035a3-d117-4936-9fb0-9f3ed354d32a-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:35.530250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.530142 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x" pod "96a035a3-d117-4936-9fb0-9f3ed354d32a" (UID: "96a035a3-d117-4936-9fb0-9f3ed354d32a"). InnerVolumeSpecName "kube-api-access-m672x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:35.673534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.672754 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m672x\" (UniqueName: \"kubernetes.io/projected/96a035a3-d117-4936-9fb0-9f3ed354d32a-kube-api-access-m672x\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:35.808584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.808551 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:08:35.820139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.820106 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" Oct 29 22:08:35.820139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.820116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8" event={"ID":"96a035a3-d117-4936-9fb0-9f3ed354d32a","Type":"ContainerDied","Data":"0355745b3bda62e6e94ccfa58945e8c8e06b31ed2956359889f2998ac81d0fbf"} Oct 29 22:08:35.820382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.820226 4376 scope.go:122] "RemoveContainer" containerID="7293e31127a9490d0c2c702a4eba744eda523cefca7eed82cc596686fd62eb5d" Oct 29 22:08:35.821757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.821727 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" event={"ID":"88eca3d2-8cfb-413a-bf90-d0b554072dc3","Type":"ContainerDied","Data":"7e886cf554bcae984f247c6a4821721b3c4b7403b9178fe4aa0f883e831868f1"} Oct 29 22:08:35.821757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.821751 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb" Oct 29 22:08:35.839019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.838988 4376 scope.go:122] "RemoveContainer" containerID="125c7d9a362ee5c0a79584f06261efb7fbdf0e0a30665b194d94ccec04709cf4" Oct 29 22:08:35.909975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.909933 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:08:35.914457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.914426 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9bb81b-7bd948856d-4qvs8"] Oct 29 22:08:35.976314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.976274 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 29 22:08:35.976530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.976340 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 29 22:08:35.979841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.979807 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\" (UniqueName: \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\") pod \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " Oct 29 22:08:35.980088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.980057 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\") pod \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\" (UID: \"88eca3d2-8cfb-413a-bf90-d0b554072dc3\") " Oct 29 22:08:35.980518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.980486 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume" pod "88eca3d2-8cfb-413a-bf90-d0b554072dc3" (UID: "88eca3d2-8cfb-413a-bf90-d0b554072dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:35.980792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.980771 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eca3d2-8cfb-413a-bf90-d0b554072dc3-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:35.982259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:35.982236 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5" pod "88eca3d2-8cfb-413a-bf90-d0b554072dc3" (UID: "88eca3d2-8cfb-413a-bf90-d0b554072dc3"). InnerVolumeSpecName "kube-api-access-9s4s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:36.082503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:36.082456 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4s5\" (UniqueName: \"kubernetes.io/projected/88eca3d2-8cfb-413a-bf90-d0b554072dc3-kube-api-access-9s4s5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:36.176715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:36.176669 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:08:36.180740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:36.180698 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9bb81b-6c6487bff-b94nb"] Oct 29 22:08:37.274775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:37.274738 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" path="/var/lib/kubelet/pods/88eca3d2-8cfb-413a-bf90-d0b554072dc3/volumes" Oct 29 22:08:37.275298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:37.275279 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a035a3-d117-4936-9fb0-9f3ed354d32a" path="/var/lib/kubelet/pods/96a035a3-d117-4936-9fb0-9f3ed354d32a/volumes" Oct 29 22:08:37.725267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:37.725222 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:08:37.725555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:37.725522 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" podUID="037b2971-dbc9-4e31-a17e-790c2ad21dab" containerName="maas-api" containerID="cri-o://90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df" gracePeriod=30 Oct 29 22:08:38.096932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.088623 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:08:38.096932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.088713 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:38.627760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.627723 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:08:38.657655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.657618 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:38.658041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.658014 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:38.659852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.659822 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:08:38.661576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.661519 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\") pod \"037b2971-dbc9-4e31-a17e-790c2ad21dab\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " Oct 29 22:08:38.661828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.661705 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\") pod \"037b2971-dbc9-4e31-a17e-790c2ad21dab\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " Oct 29 22:08:38.661828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.661803 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\" (UniqueName: \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\") pod \"037b2971-dbc9-4e31-a17e-790c2ad21dab\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " Oct 29 22:08:38.662090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.662069 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\") pod \"037b2971-dbc9-4e31-a17e-790c2ad21dab\" (UID: \"037b2971-dbc9-4e31-a17e-790c2ad21dab\") " Oct 29 22:08:38.662529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.662130 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca" pod "037b2971-dbc9-4e31-a17e-790c2ad21dab" (UID: "037b2971-dbc9-4e31-a17e-790c2ad21dab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:38.663737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.663716 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037b2971-dbc9-4e31-a17e-790c2ad21dab-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:38.668567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.668446 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls" pod "037b2971-dbc9-4e31-a17e-790c2ad21dab" (UID: "037b2971-dbc9-4e31-a17e-790c2ad21dab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:38.668567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.668497 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs" pod "037b2971-dbc9-4e31-a17e-790c2ad21dab" (UID: "037b2971-dbc9-4e31-a17e-790c2ad21dab"). InnerVolumeSpecName "kube-api-access-7j6zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:38.668567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.668539 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs" pod "037b2971-dbc9-4e31-a17e-790c2ad21dab" (UID: "037b2971-dbc9-4e31-a17e-790c2ad21dab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:38.765354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.765313 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:38.765654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.765604 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6zs\" (UniqueName: \"kubernetes.io/projected/037b2971-dbc9-4e31-a17e-790c2ad21dab-kube-api-access-7j6zs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:38.765654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.765630 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037b2971-dbc9-4e31-a17e-790c2ad21dab-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:38.896062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.892720 4376 generic.go:356] "Generic (PLEG): container finished" podID="037b2971-dbc9-4e31-a17e-790c2ad21dab" containerID="90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df" exitCode=0 Oct 29 22:08:38.906966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.903286 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" Oct 29 22:08:38.915469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.915426 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" event={"ID":"037b2971-dbc9-4e31-a17e-790c2ad21dab","Type":"ContainerDied","Data":"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df"} Oct 29 22:08:38.915760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.915738 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs" event={"ID":"037b2971-dbc9-4e31-a17e-790c2ad21dab","Type":"ContainerDied","Data":"bbb3d6b7d64144bc431115f5bf90b9c3b755deb40a59eabe05284e93e033ee40"} Oct 29 22:08:38.915886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:38.915872 4376 scope.go:122] "RemoveContainer" containerID="90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df" Oct 29 22:08:39.029422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:39.021807 4376 scope.go:122] "RemoveContainer" containerID="90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df" Oct 29 22:08:39.029422 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:08:39.025431 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df\": container with ID starting with 90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df not found: ID does not exist" containerID="90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df" Oct 29 22:08:39.029422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:39.025501 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df"} err="failed to get container status \"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df\": rpc error: code = NotFound desc = could not find container \"90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df\": container with ID starting with 90cef8c3def6c2dd96762a302c9e3d1cb72a79e7131cbac30c04bb9ac3e001df not found: ID does not exist" Oct 29 22:08:39.070340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:39.070301 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:08:39.076128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:39.076088 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9bb81b-78dd4c9f66-vwvbs"] Oct 29 22:08:39.277468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:39.277404 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037b2971-dbc9-4e31-a17e-790c2ad21dab" path="/var/lib/kubelet/pods/037b2971-dbc9-4e31-a17e-790c2ad21dab/volumes" Oct 29 22:08:41.046315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.046273 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:08:41.047133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047109 4376 cpu_manager.go:414] "containerMap: removing container" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:41.047133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047132 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f39547-4c22-4c65-8088-7519d945ac7b" containerName="istio-proxy" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047145 4376 cpu_manager.go:414] "containerMap: removing container" podUID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" containerName="payload-pre-processing" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047151 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" containerName="payload-pre-processing" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047175 4376 cpu_manager.go:414] "containerMap: removing container" podUID="96a035a3-d117-4936-9fb0-9f3ed354d32a" containerName="payload-processing" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047183 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a035a3-d117-4936-9fb0-9f3ed354d32a" containerName="payload-processing" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047194 4376 cpu_manager.go:414] "containerMap: removing container" podUID="037b2971-dbc9-4e31-a17e-790c2ad21dab" containerName="maas-api" Oct 29 22:08:41.047278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047201 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037b2971-dbc9-4e31-a17e-790c2ad21dab" containerName="maas-api" Oct 29 22:08:41.047477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047325 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eca3d2-8cfb-413a-bf90-d0b554072dc3" containerName="payload-pre-processing" Oct 29 22:08:41.047477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047341 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a035a3-d117-4936-9fb0-9f3ed354d32a" containerName="payload-processing" Oct 29 22:08:41.047477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.047350 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="037b2971-dbc9-4e31-a17e-790c2ad21dab" containerName="maas-api" Oct 29 22:08:41.051516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.051485 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.070807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.070768 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:08:41.080175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.080136 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-796d71\"/\"kube-root-ca.crt\"" Oct 29 22:08:41.080390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.080310 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-796d71\"/\"openshift-service-ca.crt\"" Oct 29 22:08:41.083127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.083095 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-796d71\"/\"default-dockercfg-wcqgq\"" Oct 29 22:08:41.083296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.083129 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-796d71\"/\"praxis-route-796d71-kserve-self-signed-certs\"" Oct 29 22:08:41.117353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.117611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117456 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.117772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117747 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.117838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117788 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.117838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117816 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg99\" (UniqueName: \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.118022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.117867 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.219674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.219631 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.219874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.219681 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg99\" (UniqueName: \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.219874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.219715 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.219874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.219788 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.220099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.219899 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.220194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.220161 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.220248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.220210 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.220248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.220219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.220428 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.220406 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.222483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.222458 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.222775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.222757 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.246821 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.246789 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg99\" (UniqueName: \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\") pod \"praxis-route-796d71-kserve-5b7dc5f65c-9bzkm\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.376932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.369883 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:41.804015 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:08:41.803975 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46d3b6ce_d0f1_4313_8e18_907fc389745a.slice/crio-0f08350e757d5d5544ae8de516c072ecc9c3c5a46cfbb259e449ae65e4ce21ea WatchSource:0}: Error finding container 0f08350e757d5d5544ae8de516c072ecc9c3c5a46cfbb259e449ae65e4ce21ea: Status 404 returned error can't find the container with id 0f08350e757d5d5544ae8de516c072ecc9c3c5a46cfbb259e449ae65e4ce21ea Oct 29 22:08:41.804160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.804135 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:08:41.920371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.920333 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" event={"ID":"46d3b6ce-d0f1-4313-8e18-907fc389745a","Type":"ContainerStarted","Data":"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8"} Oct 29 22:08:41.920558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.920385 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" event={"ID":"46d3b6ce-d0f1-4313-8e18-907fc389745a","Type":"ContainerStarted","Data":"0f08350e757d5d5544ae8de516c072ecc9c3c5a46cfbb259e449ae65e4ce21ea"} Oct 29 22:08:41.966581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:41.966516 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podStartSLOduration=0.966496118 podStartE2EDuration="966.496118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:41.966152084 +0000 UTC m=+1580.814628931" watchObservedRunningTime="2026-10-29 22:08:41.966496118 +0000 UTC m=+1580.814972965" Oct 29 22:08:42.371096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:42.371055 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:42.480219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:42.480177 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:08:42.480434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:42.480407 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" podUID="6d92849c-18bc-4694-b787-7633269ba786" containerName="maas-api" containerID="cri-o://c8b1d490e617fe139d89418d72c0f036bff4926e3bdb13a05e3431735b3c7bfb" gracePeriod=30 Oct 29 22:08:43.008332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.008295 4376 generic.go:356] "Generic (PLEG): container finished" podID="6d92849c-18bc-4694-b787-7633269ba786" containerID="c8b1d490e617fe139d89418d72c0f036bff4926e3bdb13a05e3431735b3c7bfb" exitCode=0 Oct 29 22:08:43.009641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.008544 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" event={"ID":"6d92849c-18bc-4694-b787-7633269ba786","Type":"ContainerDied","Data":"c8b1d490e617fe139d89418d72c0f036bff4926e3bdb13a05e3431735b3c7bfb"} Oct 29 22:08:43.045259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.045222 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:08:43.173436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.157712 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\") pod \"6d92849c-18bc-4694-b787-7633269ba786\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " Oct 29 22:08:43.173436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.157810 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\") pod \"6d92849c-18bc-4694-b787-7633269ba786\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " Oct 29 22:08:43.190286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.174338 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\" (UniqueName: \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\") pod \"6d92849c-18bc-4694-b787-7633269ba786\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " Oct 29 22:08:43.190286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.174795 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\") pod \"6d92849c-18bc-4694-b787-7633269ba786\" (UID: \"6d92849c-18bc-4694-b787-7633269ba786\") " Oct 29 22:08:43.190286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.180440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca" pod "6d92849c-18bc-4694-b787-7633269ba786" (UID: "6d92849c-18bc-4694-b787-7633269ba786"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:43.375751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.375707 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92849c-18bc-4694-b787-7633269ba786-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:43.378275 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.378207 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls" pod "6d92849c-18bc-4694-b787-7633269ba786" (UID: "6d92849c-18bc-4694-b787-7633269ba786"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:43.379301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.379000 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs" pod "6d92849c-18bc-4694-b787-7633269ba786" (UID: "6d92849c-18bc-4694-b787-7633269ba786"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:43.379301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.379154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z" pod "6d92849c-18bc-4694-b787-7633269ba786" (UID: "6d92849c-18bc-4694-b787-7633269ba786"). InnerVolumeSpecName "kube-api-access-ht75z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:43.476893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.476852 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht75z\" (UniqueName: \"kubernetes.io/projected/6d92849c-18bc-4694-b787-7633269ba786-kube-api-access-ht75z\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:43.476893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.476885 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:43.477197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:43.476896 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92849c-18bc-4694-b787-7633269ba786-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:44.016947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:44.016897 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" Oct 29 22:08:44.016947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:44.016930 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" event={"ID":"6d92849c-18bc-4694-b787-7633269ba786","Type":"ContainerDied","Data":"7d0e5af76b6ea2d7e60420be5b6fea978a60e3cf7750302ff2f45edebc8c99cd"} Oct 29 22:08:44.017208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:44.016994 4376 scope.go:122] "RemoveContainer" containerID="c8b1d490e617fe139d89418d72c0f036bff4926e3bdb13a05e3431735b3c7bfb" Oct 29 22:08:44.079345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:44.079303 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:08:44.092594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:44.092549 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6"] Oct 29 22:08:45.274077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:45.274040 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d92849c-18bc-4694-b787-7633269ba786" path="/var/lib/kubelet/pods/6d92849c-18bc-4694-b787-7633269ba786/volumes" Oct 29 22:08:45.756875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:45.756819 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bc531-745d867c9b-glct6" podUID="6d92849c-18bc-4694-b787-7633269ba786" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.127:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:46.437351 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:08:46.437312 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:08:51.289841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.289793 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:08:51.290638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.290617 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6d92849c-18bc-4694-b787-7633269ba786" containerName="maas-api" Oct 29 22:08:51.290638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.290639 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d92849c-18bc-4694-b787-7633269ba786" containerName="maas-api" Oct 29 22:08:51.290775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.290766 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d92849c-18bc-4694-b787-7633269ba786" containerName="maas-api" Oct 29 22:08:51.296032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.295998 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.315126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.315089 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:08:51.324731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.324697 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c53cc8\"" Oct 29 22:08:51.324731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.324714 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c53cc8-dockercfg-kcxzq\"" Oct 29 22:08:51.372282 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.372240 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.372641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.372611 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.375844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.375808 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:51.479103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.475514 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.479103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.475651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.479103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.478055 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.587073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.586975 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:08:51.593332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.593297 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.625219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.622226 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.661936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.648029 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:08:51.725543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.724323 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:51.791117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.791077 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.791341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.791279 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skb7\" (UniqueName: \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.892789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.892692 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.892998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.892829 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skb7\" (UniqueName: \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.893521 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.893501 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.912461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.912425 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:51.919963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.919933 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skb7\" (UniqueName: \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\") pod \"payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:51.994119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:51.994078 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:52.453515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:52.451988 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:08:52.571218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:52.571180 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:08:52.619178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:52.619129 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:08:53.474829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.474776 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" event={"ID":"55094201-bfb3-48e8-ad02-3fb37edcdb6b","Type":"ContainerStarted","Data":"1b61ff9ea1fe32c29712ba8afa728e2b72fc776f2d6085cae5b7673d227f2e46"} Oct 29 22:08:53.475356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.474846 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" event={"ID":"55094201-bfb3-48e8-ad02-3fb37edcdb6b","Type":"ContainerStarted","Data":"10893deab3082f50fc790432c91e41cfee4e149df523f8bac5068b22e98bdd46"} Oct 29 22:08:53.475356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.474886 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:08:53.476538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.476507 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" event={"ID":"5abf1c0f-c43e-4c90-96f3-7a3417992098","Type":"ContainerStarted","Data":"05e71a6340216a6c9d90300a42b8f1f92c52d92c54aad926691bfb8a337010b3"} Oct 29 22:08:53.476706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.476562 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" event={"ID":"5abf1c0f-c43e-4c90-96f3-7a3417992098","Type":"ContainerStarted","Data":"e98f4f6f2f73172c42c3f458203163c023bc69705ddfddfc4ce45fa11f7bf385"} Oct 29 22:08:53.476784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.476750 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:08:53.524380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.524322 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" podStartSLOduration=2.52430486 podStartE2EDuration="2.52430486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:53.520516661 +0000 UTC m=+1592.368993509" watchObservedRunningTime="2026-10-29 22:08:53.52430486 +0000 UTC m=+1592.372781705" Oct 29 22:08:53.568223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:08:53.568169 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" podStartSLOduration=2.568154039 podStartE2EDuration="2.568154039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:53.565103597 +0000 UTC m=+1592.413580446" watchObservedRunningTime="2026-10-29 22:08:53.568154039 +0000 UTC m=+1592.416630884" Oct 29 22:09:03.191387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:03.191350 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 22:09:03.191973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:03.191567 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" containerID="cri-o://8266e0323206698da35084cf2c36f297e916d2e5a10f2608a61a3e90eb3b9e00" gracePeriod=30 Oct 29 22:09:04.509117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:04.509081 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:09:04.510100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:04.510076 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:09:04.989232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:04.989193 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:09:04.994682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:04.994649 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:04.996455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:04.996429 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:09:05.008814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.008783 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.010788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.010756 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:09:05.030522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.030488 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:09:05.039436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.039398 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.039978 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040074 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040133 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040193 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040267 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040344 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040385 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040453 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040545 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040609 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040635 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfqx\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040787 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnl4\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.040847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040810 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.040847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040836 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.041129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.040997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.041129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.041018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.041129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.041049 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142009 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142047 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142093 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142219 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142259 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142292 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142345 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142403 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142458 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142540 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142570 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142622 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142658 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142664 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142722 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfqx\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.142771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142745 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142798 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.143173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142841 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnl4\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.143173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142927 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.143289 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.142621 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.143565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.143540 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.143695 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.143775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.143719 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.144050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.144021 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.145053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.145027 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.145632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.145603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.145735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.145673 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.145735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.145729 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.168633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.168597 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfqx\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.168777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.168717 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.168777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.168736 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnl4\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.168939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.168917 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.346447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.344971 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:05.360926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.353327 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:05.747056 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.747021 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:09:05.747578 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:09:05.747551 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b453d_cce6_4ca8_9133_ab91353f1249.slice/crio-1d542a58fceeb6029c50dd96ae2766251f9d7ae60703a01a11d85cfdfb91ecd1 WatchSource:0}: Error finding container 1d542a58fceeb6029c50dd96ae2766251f9d7ae60703a01a11d85cfdfb91ecd1: Status 404 returned error can't find the container with id 1d542a58fceeb6029c50dd96ae2766251f9d7ae60703a01a11d85cfdfb91ecd1 Oct 29 22:09:05.750788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.750748 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:05.750941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.750821 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:05.750941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.750851 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:05.759715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.759688 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:09:05.762427 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:09:05.762399 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa24378_8958_4bbb_818f_bc36e45aca27.slice/crio-01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb WatchSource:0}: Error finding container 01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb: Status 404 returned error can't find the container with id 01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb Oct 29 22:09:05.765106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.765070 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:05.765187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.765143 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:05.765187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:05.765171 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:09:06.610674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.610634 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" event={"ID":"7aa24378-8958-4bbb-818f-bc36e45aca27","Type":"ContainerStarted","Data":"f59613b123104fedd40a7ae773a5c28bf3c50e6bf290bf0cb3746158ea7c4720"} Oct 29 22:09:06.610883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.610703 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" event={"ID":"7aa24378-8958-4bbb-818f-bc36e45aca27","Type":"ContainerStarted","Data":"01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb"} Oct 29 22:09:06.612406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.612375 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" event={"ID":"f89b453d-cce6-4ca8-9133-ab91353f1249","Type":"ContainerStarted","Data":"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f"} Oct 29 22:09:06.612534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.612435 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" event={"ID":"f89b453d-cce6-4ca8-9133-ab91353f1249","Type":"ContainerStarted","Data":"1d542a58fceeb6029c50dd96ae2766251f9d7ae60703a01a11d85cfdfb91ecd1"} Oct 29 22:09:06.751929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.747949 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podStartSLOduration=2.747926487 podStartE2EDuration="2.747926487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.694335915 +0000 UTC m=+1605.542812763" watchObservedRunningTime="2026-10-29 22:09:06.747926487 +0000 UTC m=+1605.596403332" Oct 29 22:09:06.764926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:06.760869 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podStartSLOduration=2.760843849 podStartE2EDuration="2.760843849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.745225532 +0000 UTC m=+1605.593702379" watchObservedRunningTime="2026-10-29 22:09:06.760843849 +0000 UTC m=+1605.609320696" Oct 29 22:09:07.356927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.356505 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:07.356927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.356585 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:07.360927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.360041 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:07.360927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.360120 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:07.360927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.360423 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:07.360927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:07.360455 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:08.356929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.356418 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:08.356929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.356484 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:08.357505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.356945 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:08.357505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.356982 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:08.631724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.631687 4376 generic.go:356] "Generic (PLEG): container finished" podID="15a71abd-5771-4c76-a767-54ff21dad866" containerID="8266e0323206698da35084cf2c36f297e916d2e5a10f2608a61a3e90eb3b9e00" exitCode=0 Oct 29 22:09:08.631929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.631762 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" event={"ID":"15a71abd-5771-4c76-a767-54ff21dad866","Type":"ContainerDied","Data":"8266e0323206698da35084cf2c36f297e916d2e5a10f2608a61a3e90eb3b9e00"} Oct 29 22:09:08.631929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.631858 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" event={"ID":"15a71abd-5771-4c76-a767-54ff21dad866","Type":"ContainerDied","Data":"06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f"} Oct 29 22:09:08.631929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.631885 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06aa3646a44bee1dd1cfd867a7caac6ce51e67869b2fa5ff9a8d405b6cda725f" Oct 29 22:09:08.689409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.689374 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 22:09:08.805238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805199 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805257 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805503 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805633 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805672 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805701 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:08.805813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805718 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.805813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805766 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:08.806104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.805914 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\") pod \"15a71abd-5771-4c76-a767-54ff21dad866\" (UID: \"15a71abd-5771-4c76-a767-54ff21dad866\") " Oct 29 22:09:08.806104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.806016 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:08.806581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.806509 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:08.806820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.806801 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.806999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.806983 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.807342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.807320 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.807699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.807673 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:08.811027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.810852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "kube-api-access-fd44p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:08.811581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.811557 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:08.811870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.811840 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:08.812718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.812693 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo" pod "15a71abd-5771-4c76-a767-54ff21dad866" (UID: "15a71abd-5771-4c76-a767-54ff21dad866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914461 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a71abd-5771-4c76-a767-54ff21dad866-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914512 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd44p\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-kube-api-access-fd44p\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914530 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a71abd-5771-4c76-a767-54ff21dad866-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914544 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914570 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a71abd-5771-4c76-a767-54ff21dad866-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:08.914994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:08.914583 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a71abd-5771-4c76-a767-54ff21dad866-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:09.355962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.355486 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:09.355962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.355552 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:09.388964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.382994 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:09.389406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.389056 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:09.689261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.689175 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv" Oct 29 22:09:09.785096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.785058 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 22:09:09.791199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:09.791164 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-tfbjv"] Oct 29 22:09:10.354930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:10.352086 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:10.354930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:10.352149 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:10.365936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:10.364365 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:10.365936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:10.364428 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:11.291095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:11.291057 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a71abd-5771-4c76-a767-54ff21dad866" path="/var/lib/kubelet/pods/15a71abd-5771-4c76-a767-54ff21dad866/volumes" Oct 29 22:09:11.354927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:11.354339 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:11.354927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:11.354414 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:11.354927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:11.354536 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:11.354927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:11.354580 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:12.356399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:12.356334 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:12.356983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:12.356425 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:12.357124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:12.357089 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:12.357210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:12.357138 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:13.348582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:13.348545 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 29 22:09:13.348781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:13.348606 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 29 22:09:13.354045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:13.354012 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:09:13.354172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:13.354073 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 29 22:09:14.354553 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.354512 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:14.355208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.355045 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:14.355991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.355969 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:09:14.379485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.379453 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:14.379823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.379801 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:14.381233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:14.381197 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:09:16.436739 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:09:16.436698 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:09:17.405921 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:17.405730 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:09:17.406122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:17.405992 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" podUID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" containerName="maas-api" containerID="cri-o://b261100780904d3868b83c1352a4c4ff44af8c7724bcf209c5d3bec6737310e6" gracePeriod=30 Oct 29 22:09:17.989976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:17.989750 4376 generic.go:356] "Generic (PLEG): container finished" podID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" containerID="b261100780904d3868b83c1352a4c4ff44af8c7724bcf209c5d3bec6737310e6" exitCode=0 Oct 29 22:09:17.989976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:17.989861 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" event={"ID":"63b6cccb-2039-46d1-8843-0cd1fd5ce95f","Type":"ContainerDied","Data":"b261100780904d3868b83c1352a4c4ff44af8c7724bcf209c5d3bec6737310e6"} Oct 29 22:09:18.100974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.100083 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:09:18.238843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.238439 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\") pod \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " Oct 29 22:09:18.238843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.238683 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\") pod \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " Oct 29 22:09:18.238843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.238849 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\") pod \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " Oct 29 22:09:18.239197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.238924 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\" (UniqueName: \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\") pod \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\" (UID: \"63b6cccb-2039-46d1-8843-0cd1fd5ce95f\") " Oct 29 22:09:18.240395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.240365 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca" pod "63b6cccb-2039-46d1-8843-0cd1fd5ce95f" (UID: "63b6cccb-2039-46d1-8843-0cd1fd5ce95f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:18.243127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.243093 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs" pod "63b6cccb-2039-46d1-8843-0cd1fd5ce95f" (UID: "63b6cccb-2039-46d1-8843-0cd1fd5ce95f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:18.245562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.245509 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls" pod "63b6cccb-2039-46d1-8843-0cd1fd5ce95f" (UID: "63b6cccb-2039-46d1-8843-0cd1fd5ce95f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:18.246389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.246357 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:18.246503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.246391 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:18.246503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.246436 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:18.252934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.252866 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh" pod "63b6cccb-2039-46d1-8843-0cd1fd5ce95f" (UID: "63b6cccb-2039-46d1-8843-0cd1fd5ce95f"). InnerVolumeSpecName "kube-api-access-bj2bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:18.375884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.375808 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2bh\" (UniqueName: \"kubernetes.io/projected/63b6cccb-2039-46d1-8843-0cd1fd5ce95f-kube-api-access-bj2bh\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:18.999534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.999497 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" Oct 29 22:09:18.999534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.999512 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999" event={"ID":"63b6cccb-2039-46d1-8843-0cd1fd5ce95f","Type":"ContainerDied","Data":"6f50ff9efb32e0cc795a9bd822764fd08850b702c71b2db00bbf88beb87f8eb5"} Oct 29 22:09:19.000219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:18.999596 4376 scope.go:122] "RemoveContainer" containerID="b261100780904d3868b83c1352a4c4ff44af8c7724bcf209c5d3bec6737310e6" Oct 29 22:09:19.061266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:19.061226 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:09:19.074533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:19.074501 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d9498-6f49f8c4f8-xm999"] Oct 29 22:09:19.273845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:19.273771 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" path="/var/lib/kubelet/pods/63b6cccb-2039-46d1-8843-0cd1fd5ce95f/volumes" Oct 29 22:09:23.532956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:23.532925 4376 scope.go:122] "RemoveContainer" containerID="846b8516867e03f812835a97f2483c24fdefa8c69830b45de5c8f36542236dea" Oct 29 22:09:23.544055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:23.544032 4376 scope.go:122] "RemoveContainer" containerID="8266e0323206698da35084cf2c36f297e916d2e5a10f2608a61a3e90eb3b9e00" Oct 29 22:09:41.835705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:41.835671 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:09:41.836374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:41.835953 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" podUID="faf65796-bd09-45e0-a101-6e9845d34555" containerName="maas-api" containerID="cri-o://3599daa2fbb6907401a3f31e3e7a0f4c87c0dddd64adef7e0c59433a35613d53" gracePeriod=30 Oct 29 22:09:42.422748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.422701 4376 generic.go:356] "Generic (PLEG): container finished" podID="faf65796-bd09-45e0-a101-6e9845d34555" containerID="3599daa2fbb6907401a3f31e3e7a0f4c87c0dddd64adef7e0c59433a35613d53" exitCode=0 Oct 29 22:09:42.423264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.423061 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" event={"ID":"faf65796-bd09-45e0-a101-6e9845d34555","Type":"ContainerDied","Data":"3599daa2fbb6907401a3f31e3e7a0f4c87c0dddd64adef7e0c59433a35613d53"} Oct 29 22:09:42.428925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.423722 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:09:42.440343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.432406 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" event={"ID":"faf65796-bd09-45e0-a101-6e9845d34555","Type":"ContainerDied","Data":"4bf069be821396dcded760c60060284921c3b97d67898aa52b5bc304673cae48"} Oct 29 22:09:42.440343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.432513 4376 scope.go:122] "RemoveContainer" containerID="3599daa2fbb6907401a3f31e3e7a0f4c87c0dddd64adef7e0c59433a35613d53" Oct 29 22:09:42.565757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.565713 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\") pod \"faf65796-bd09-45e0-a101-6e9845d34555\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.574662 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\") pod \"faf65796-bd09-45e0-a101-6e9845d34555\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.574744 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") pod \"faf65796-bd09-45e0-a101-6e9845d34555\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.574976 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\" (UniqueName: \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\") pod \"faf65796-bd09-45e0-a101-6e9845d34555\" (UID: \"faf65796-bd09-45e0-a101-6e9845d34555\") " Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.574443 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca" pod "faf65796-bd09-45e0-a101-6e9845d34555" (UID: "faf65796-bd09-45e0-a101-6e9845d34555"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.580824 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs" pod "faf65796-bd09-45e0-a101-6e9845d34555" (UID: "faf65796-bd09-45e0-a101-6e9845d34555"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:42.584936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.583126 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls" pod "faf65796-bd09-45e0-a101-6e9845d34555" (UID: "faf65796-bd09-45e0-a101-6e9845d34555"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:42.618961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.617877 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79" pod "faf65796-bd09-45e0-a101-6e9845d34555" (UID: "faf65796-bd09-45e0-a101-6e9845d34555"). InnerVolumeSpecName "kube-api-access-qrd79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:42.679248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.679207 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf65796-bd09-45e0-a101-6e9845d34555-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:42.679248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.679241 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:42.679248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.679252 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf65796-bd09-45e0-a101-6e9845d34555-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:42.679503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:42.679260 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrd79\" (UniqueName: \"kubernetes.io/projected/faf65796-bd09-45e0-a101-6e9845d34555-kube-api-access-qrd79\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:43.432488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:43.432455 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv" Oct 29 22:09:43.493013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:43.492980 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:09:43.509631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:43.509595 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6153f8-88754995-qlbvv"] Oct 29 22:09:45.274652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:45.274606 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf65796-bd09-45e0-a101-6e9845d34555" path="/var/lib/kubelet/pods/faf65796-bd09-45e0-a101-6e9845d34555/volumes" Oct 29 22:09:46.523750 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:09:46.523707 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:09:48.253930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:48.248840 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:09:48.372010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:48.371470 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" containerID="cri-o://15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636" gracePeriod=30 Oct 29 22:09:53.064265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:53.064221 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5_b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a/payload-processing/0.log" Oct 29 22:09:53.346834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:53.346743 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:09:53.653459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:53.653382 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:09:54.063398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.063372 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:09:54.200874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.200844 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201056 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201184 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201250 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201176 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201324 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:54.201399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201447 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201512 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201579 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\") pod \"640793d4-5e74-4952-a98d-d81df4d6a17d\" (UID: \"640793d4-5e74-4952-a98d-d81df4d6a17d\") " Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201633 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201644 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:54.201726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.201669 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:54.202291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.202268 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.202388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.202292 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640793d4-5e74-4952-a98d-d81df4d6a17d-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.202388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.202305 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.202388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.202317 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.202388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.202330 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.203626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.203602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:09:54.203770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.203751 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:54.203998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.203983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:54.204059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.204019 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q" pod "640793d4-5e74-4952-a98d-d81df4d6a17d" (UID: "640793d4-5e74-4952-a98d-d81df4d6a17d"). InnerVolumeSpecName "kube-api-access-9zj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:54.303593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.303559 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.303593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.303589 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zj2q\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-kube-api-access-9zj2q\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.303593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.303600 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.303837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.303608 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640793d4-5e74-4952-a98d-d81df4d6a17d-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.657760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.657667 4376 generic.go:356] "Generic (PLEG): container finished" podID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerID="15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636" exitCode=0 Oct 29 22:09:54.657760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.657716 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" Oct 29 22:09:54.657760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.657751 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerDied","Data":"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636"} Oct 29 22:09:54.658058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.657806 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2" event={"ID":"640793d4-5e74-4952-a98d-d81df4d6a17d","Type":"ContainerDied","Data":"384b97292ef70858b35b619084611d5ab92f3eeb4b106ca7537a70b4fb19b4d8"} Oct 29 22:09:54.658058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.657834 4376 scope.go:122] "RemoveContainer" containerID="15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636" Oct 29 22:09:54.669895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.669874 4376 scope.go:122] "RemoveContainer" containerID="b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc" Oct 29 22:09:54.680360 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.680337 4376 scope.go:122] "RemoveContainer" containerID="15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636" Oct 29 22:09:54.680672 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:09:54.680647 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636\": container with ID starting with 15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636 not found: ID does not exist" containerID="15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636" Oct 29 22:09:54.680759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.680682 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636"} err="failed to get container status \"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636\": rpc error: code = NotFound desc = could not find container \"15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636\": container with ID starting with 15448e5c93460d4dbc8f8a31af2887af39a9d5e59f34462bbafc2ca631636636 not found: ID does not exist" Oct 29 22:09:54.680759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.680707 4376 scope.go:122] "RemoveContainer" containerID="b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc" Oct 29 22:09:54.681064 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:09:54.681040 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc\": container with ID starting with b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc not found: ID does not exist" containerID="b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc" Oct 29 22:09:54.681171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.681075 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc"} err="failed to get container status \"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc\": rpc error: code = NotFound desc = could not find container \"b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc\": container with ID starting with b63baf1705a499be571c2b31483bc6f3627d26adaaa55b8c08ba192b97541afc not found: ID does not exist" Oct 29 22:09:54.733760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.733730 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:09:54.746454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:54.746429 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fd1d4e-openshift-default-c6bf8759-htbl2"] Oct 29 22:09:55.278421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:09:55.278386 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" path="/var/lib/kubelet/pods/640793d4-5e74-4952-a98d-d81df4d6a17d/volumes" Oct 29 22:10:06.978940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:06.978840 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 22:10:06.979343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:06.979116 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" containerID="cri-o://e37c3bb82f4b0cb501eb410f5221a512473cc7b4a3adc3fa4d49800999471679" gracePeriod=30 Oct 29 22:10:10.977212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:10.977166 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:10:10.977759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:10.977432 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerName="payload-processing" containerID="cri-o://1b61ff9ea1fe32c29712ba8afa728e2b72fc776f2d6085cae5b7673d227f2e46" gracePeriod=30 Oct 29 22:10:10.992285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:10.992247 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:10:10.992563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:10.992507 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" podUID="5abf1c0f-c43e-4c90-96f3-7a3417992098" containerName="payload-pre-processing" containerID="cri-o://05e71a6340216a6c9d90300a42b8f1f92c52d92c54aad926691bfb8a337010b3" gracePeriod=30 Oct 29 22:10:12.450225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:12.450181 4376 generic.go:356] "Generic (PLEG): container finished" podID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerID="1b61ff9ea1fe32c29712ba8afa728e2b72fc776f2d6085cae5b7673d227f2e46" exitCode=0 Oct 29 22:10:12.451038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:12.451005 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" event={"ID":"55094201-bfb3-48e8-ad02-3fb37edcdb6b","Type":"ContainerDied","Data":"1b61ff9ea1fe32c29712ba8afa728e2b72fc776f2d6085cae5b7673d227f2e46"} Oct 29 22:10:12.473101 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:12.473057 4376 generic.go:356] "Generic (PLEG): container finished" podID="5abf1c0f-c43e-4c90-96f3-7a3417992098" containerID="05e71a6340216a6c9d90300a42b8f1f92c52d92c54aad926691bfb8a337010b3" exitCode=0 Oct 29 22:10:12.473406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:12.473380 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" event={"ID":"5abf1c0f-c43e-4c90-96f3-7a3417992098","Type":"ContainerDied","Data":"05e71a6340216a6c9d90300a42b8f1f92c52d92c54aad926691bfb8a337010b3"} Oct 29 22:10:13.094932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.092003 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.101378 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.101980 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.104322 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.132641 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.133622 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.133845 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 29 22:10:13.151236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.133888 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:10:13.192937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.188942 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.841542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.838725 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.965922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:13.965360 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.114930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.094961 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:10:14.186814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.186756 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.192931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.189734 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": context deadline exceeded" Oct 29 22:10:14.193563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.193524 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:10:14.325990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.324641 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.325990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.324918 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:10:14.325990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.325202 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:14.426641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.426590 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:14.427568 4376 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.132s" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427654 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427706 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427742 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427775 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427810 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427845 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427899 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:10:14.441990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.427955 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:10:14.522232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.522031 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:10:14.523834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.523796 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" event={"ID":"5abf1c0f-c43e-4c90-96f3-7a3417992098","Type":"ContainerDied","Data":"e98f4f6f2f73172c42c3f458203163c023bc69705ddfddfc4ce45fa11f7bf385"} Oct 29 22:10:14.523991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.523871 4376 scope.go:122] "RemoveContainer" containerID="05e71a6340216a6c9d90300a42b8f1f92c52d92c54aad926691bfb8a337010b3" Oct 29 22:10:14.530628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.530505 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:10:14.534438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.534413 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:10:14.534573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.534475 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:10:14.535111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.535083 4376 generic.go:356] "Generic (PLEG): container finished" podID="2d26d931-118e-4738-9aaa-a0074161de15" containerID="e37c3bb82f4b0cb501eb410f5221a512473cc7b4a3adc3fa4d49800999471679" exitCode=0 Oct 29 22:10:14.542290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.542267 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:10:14.542658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.542624 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerDied","Data":"e37c3bb82f4b0cb501eb410f5221a512473cc7b4a3adc3fa4d49800999471679"} Oct 29 22:10:14.545894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.545849 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.546953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.546611 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:10:14.557010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.556982 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:10:14.582431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.581523 4376 scope.go:122] "RemoveContainer" containerID="db58965ebd58993603641e2682d9ab0691c5f758d4f072466b9875f81982b925" Oct 29 22:10:14.584276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.584253 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:10:14.590444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.590417 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:10:14.599704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.594213 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 22:10:14.602628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.602608 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:10:14.648055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648023 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\") pod \"5abf1c0f-c43e-4c90-96f3-7a3417992098\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " Oct 29 22:10:14.648296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648065 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\") pod \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " Oct 29 22:10:14.648296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648205 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\" (UniqueName: \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\") pod \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\" (UID: \"55094201-bfb3-48e8-ad02-3fb37edcdb6b\") " Oct 29 22:10:14.648296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648268 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648363 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648417 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\") pod \"5abf1c0f-c43e-4c90-96f3-7a3417992098\" (UID: \"5abf1c0f-c43e-4c90-96f3-7a3417992098\") " Oct 29 22:10:14.648597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648509 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648614 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648647 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648770 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648802 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume" pod "55094201-bfb3-48e8-ad02-3fb37edcdb6b" (UID: "55094201-bfb3-48e8-ad02-3fb37edcdb6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:14.648855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648834 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.648983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.648969 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.649042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649018 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\") pod \"2d26d931-118e-4738-9aaa-a0074161de15\" (UID: \"2d26d931-118e-4738-9aaa-a0074161de15\") " Oct 29 22:10:14.649110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649085 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume" pod "5abf1c0f-c43e-4c90-96f3-7a3417992098" (UID: "5abf1c0f-c43e-4c90-96f3-7a3417992098"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:14.649487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649461 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:14.649738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:14.649856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649819 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:14.649978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.649927 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:14.650178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650153 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:14.650260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650181 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55094201-bfb3-48e8-ad02-3fb37edcdb6b-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.650260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650205 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abf1c0f-c43e-4c90-96f3-7a3417992098-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.650260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650221 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.650260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650238 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.650260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650253 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d26d931-118e-4738-9aaa-a0074161de15-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.650563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.650268 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.652543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.652496 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:10:14.652944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.652889 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "kube-api-access-svtt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:14.653147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.653128 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:14.653519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.653497 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs" pod "5abf1c0f-c43e-4c90-96f3-7a3417992098" (UID: "5abf1c0f-c43e-4c90-96f3-7a3417992098"). InnerVolumeSpecName "kube-api-access-t5hgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:14.653682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.653662 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy" pod "2d26d931-118e-4738-9aaa-a0074161de15" (UID: "2d26d931-118e-4738-9aaa-a0074161de15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:14.656053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.656011 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7" pod "55094201-bfb3-48e8-ad02-3fb37edcdb6b" (UID: "55094201-bfb3-48e8-ad02-3fb37edcdb6b"). InnerVolumeSpecName "kube-api-access-8skb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:14.752822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752778 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.752822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752827 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/5abf1c0f-c43e-4c90-96f3-7a3417992098-kube-api-access-t5hgs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.753059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752843 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skb7\" (UniqueName: \"kubernetes.io/projected/55094201-bfb3-48e8-ad02-3fb37edcdb6b-kube-api-access-8skb7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.753059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752860 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d26d931-118e-4738-9aaa-a0074161de15-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.753059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752875 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.753059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752890 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d26d931-118e-4738-9aaa-a0074161de15-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:14.753059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:14.752920 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtt6\" (UniqueName: \"kubernetes.io/projected/2d26d931-118e-4738-9aaa-a0074161de15-kube-api-access-svtt6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:15.509853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.509761 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.138:9004: i/o timeout" Oct 29 22:10:15.560252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.560217 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" event={"ID":"2d26d931-118e-4738-9aaa-a0074161de15","Type":"ContainerDied","Data":"417a6882ccadce1c83840ea212859e033e5dd7d82ecd7d799b05b17a3cac5a3a"} Oct 29 22:10:15.560445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.560273 4376 scope.go:122] "RemoveContainer" containerID="e37c3bb82f4b0cb501eb410f5221a512473cc7b4a3adc3fa4d49800999471679" Oct 29 22:10:15.560445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.560378 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6" Oct 29 22:10:15.563262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.563228 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" event={"ID":"55094201-bfb3-48e8-ad02-3fb37edcdb6b","Type":"ContainerDied","Data":"10893deab3082f50fc790432c91e41cfee4e149df523f8bac5068b22e98bdd46"} Oct 29 22:10:15.564302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.564274 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8" Oct 29 22:10:15.570185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.570109 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh" Oct 29 22:10:15.577073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.577039 4376 scope.go:122] "RemoveContainer" containerID="1b61ff9ea1fe32c29712ba8afa728e2b72fc776f2d6085cae5b7673d227f2e46" Oct 29 22:10:15.661503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.661469 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 22:10:15.666626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.666595 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-jmqp6"] Oct 29 22:10:15.713344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.713310 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:10:15.716816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.716795 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-5f8455496c-vbvkh"] Oct 29 22:10:15.750956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.750879 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:10:15.755807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:15.755782 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-95b59d778-vp9w8"] Oct 29 22:10:16.435690 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:16.435649 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:10:17.276290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:17.276254 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d26d931-118e-4738-9aaa-a0074161de15" path="/var/lib/kubelet/pods/2d26d931-118e-4738-9aaa-a0074161de15/volumes" Oct 29 22:10:17.276770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:17.276754 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" path="/var/lib/kubelet/pods/55094201-bfb3-48e8-ad02-3fb37edcdb6b/volumes" Oct 29 22:10:17.277188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:17.277172 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abf1c0f-c43e-4c90-96f3-7a3417992098" path="/var/lib/kubelet/pods/5abf1c0f-c43e-4c90-96f3-7a3417992098/volumes" Oct 29 22:10:18.241109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241078 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:10:18.241772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241757 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.241818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241774 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.241818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241788 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.241818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241793 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.241818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241806 4376 cpu_manager.go:414] "containerMap: removing container" podUID="faf65796-bd09-45e0-a101-6e9845d34555" containerName="maas-api" Oct 29 22:10:18.241818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241813 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf65796-bd09-45e0-a101-6e9845d34555" containerName="maas-api" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241832 4376 cpu_manager.go:414] "containerMap: removing container" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241836 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241849 4376 cpu_manager.go:414] "containerMap: removing container" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241853 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241862 4376 cpu_manager.go:414] "containerMap: removing container" podUID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" containerName="maas-api" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241866 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" containerName="maas-api" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241883 4376 cpu_manager.go:414] "containerMap: removing container" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerName="payload-processing" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241887 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerName="payload-processing" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241895 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5abf1c0f-c43e-4c90-96f3-7a3417992098" containerName="payload-pre-processing" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241916 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abf1c0f-c43e-4c90-96f3-7a3417992098" containerName="payload-pre-processing" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241933 4376 cpu_manager.go:414] "containerMap: removing container" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.241938 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" Oct 29 22:10:18.242045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242044 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="55094201-bfb3-48e8-ad02-3fb37edcdb6b" containerName="payload-processing" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242054 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abf1c0f-c43e-4c90-96f3-7a3417992098" containerName="payload-pre-processing" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242062 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a71abd-5771-4c76-a767-54ff21dad866" containerName="istio-proxy" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242070 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242076 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d26d931-118e-4738-9aaa-a0074161de15" containerName="istio-proxy" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242081 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf65796-bd09-45e0-a101-6e9845d34555" containerName="maas-api" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242090 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242098 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="640793d4-5e74-4952-a98d-d81df4d6a17d" containerName="istio-proxy" Oct 29 22:10:18.242436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.242104 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b6cccb-2039-46d1-8843-0cd1fd5ce95f" containerName="maas-api" Oct 29 22:10:18.247737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.247715 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.289991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290159 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290334 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbmx\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290361 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290556 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290584 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290724 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290823 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.290984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.290871 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.291787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.291763 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:10:18.392324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392330 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392403 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392514 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392574 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392655 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392718 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbmx\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392732 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392749 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392753 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.392975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.392878 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.393187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.393168 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.393345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.393324 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.394764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.394743 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.395054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.395036 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.418414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.418370 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.418542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.418522 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbmx\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.568310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.568211 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:18.896325 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:10:18.896296 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7275c8f6_719f_43b4_b7e9_a50b89309a51.slice/crio-cbc28cb51ab988cf102e09fea80a35db8c58a9ca134c426e161e542a55aff2bd WatchSource:0}: Error finding container cbc28cb51ab988cf102e09fea80a35db8c58a9ca134c426e161e542a55aff2bd: Status 404 returned error can't find the container with id cbc28cb51ab988cf102e09fea80a35db8c58a9ca134c426e161e542a55aff2bd Oct 29 22:10:18.896695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.896668 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:10:18.900115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.899348 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:18.900115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.899614 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:18.900115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:18.899665 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:19.606271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:19.606232 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" event={"ID":"7275c8f6-719f-43b4-b7e9-a50b89309a51","Type":"ContainerStarted","Data":"296218c5be44c24eecdceab37420e4aa4cfdb4af2a2060bfb408ac10d9379632"} Oct 29 22:10:19.606692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:19.606292 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" event={"ID":"7275c8f6-719f-43b4-b7e9-a50b89309a51","Type":"ContainerStarted","Data":"cbc28cb51ab988cf102e09fea80a35db8c58a9ca134c426e161e542a55aff2bd"} Oct 29 22:10:19.670060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:19.670000 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podStartSLOduration=1.6699791130000001 podStartE2EDuration="1.669979113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:19.668916216 +0000 UTC m=+1678.517393054" watchObservedRunningTime="2026-10-29 22:10:19.669979113 +0000 UTC m=+1678.518455965" Oct 29 22:10:20.573957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:20.572689 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:20.620928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:20.611462 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:20.620928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:20.611524 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:21.570971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:21.570192 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:21.570971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:21.570268 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:22.430140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.430109 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:10:22.454233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.454202 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.472807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.472775 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:10:22.473988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.473967 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-80bd36-serving-cert\"" Oct 29 22:10:22.526464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.526425 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcw9w\" (UniqueName: \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.526670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.526549 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.526670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.526587 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.526670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.526665 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.570308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.570273 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:22.570599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.570570 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:22.666929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.630208 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcw9w\" (UniqueName: \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.667318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.667286 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.667471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.667456 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.667697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.667680 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.669203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.669176 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.669473 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:22.669456 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-serving-cert: secret "maas-api-e2e-praxis-a-80bd36-serving-cert" not found Oct 29 22:10:22.669631 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:22.669620 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls podName:a294798d-a43a-40e7-b398-3e6132e976f9 nodeName:}" failed. No retries permitted until 2026-10-29 22:10:23.169597711 +0000 UTC m=+1682.018074549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls") pod "maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" (UID: "a294798d-a43a-40e7-b398-3e6132e976f9") : secret "maas-api-e2e-praxis-a-80bd36-serving-cert" not found Oct 29 22:10:22.716003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.715967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.720535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.720456 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcw9w\" (UniqueName: \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:22.890164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.890135 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:10:22.897955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.897349 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.918050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.918025 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7721b3-openshift-default-dockercfg-5bxz7\"" Oct 29 22:10:22.922675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.922638 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:10:22.976511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976423 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.976691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976566 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.976691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976632 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.976823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.976823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976766 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcdk\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.976823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.977034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.976997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.977214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.977188 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:22.977299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:22.977238 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.093117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.093068 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.093475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.093453 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.093625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.093607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.093800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.093784 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.098941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.094073 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.099288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.099253 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.099556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.099537 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.099731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.099716 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcdk\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.099855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.099840 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.121932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.100584 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.123100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.123066 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.194268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.124972 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.220276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.125216 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.220983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.205651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:23.226840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.220157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.249013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.248932 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.249690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.249634 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.260997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.260921 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:23.272957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.272849 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcdk\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.341943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.340810 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\") pod \"e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.390486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.389302 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:23.533104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.531344 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:23.572147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.572076 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:23.572446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:23.572393 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:24.170650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.170614 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:10:24.172182 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:10:24.172155 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda294798d_a43a_40e7_b398_3e6132e976f9.slice/crio-6f4654c4a07d95d3b16eea1099bc7430e2c87ca5f6c88f14467d7a304f8a1757 WatchSource:0}: Error finding container 6f4654c4a07d95d3b16eea1099bc7430e2c87ca5f6c88f14467d7a304f8a1757: Status 404 returned error can't find the container with id 6f4654c4a07d95d3b16eea1099bc7430e2c87ca5f6c88f14467d7a304f8a1757 Oct 29 22:10:24.186435 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:10:24.186395 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b98a77c_450d_4a4e_b812_ad4f1dd81af5.slice/crio-b3251c1d64a227e276cbc450be05285187d0239c5a0095747b70b5a5efee3bf8 WatchSource:0}: Error finding container b3251c1d64a227e276cbc450be05285187d0239c5a0095747b70b5a5efee3bf8: Status 404 returned error can't find the container with id b3251c1d64a227e276cbc450be05285187d0239c5a0095747b70b5a5efee3bf8 Oct 29 22:10:24.187341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.187314 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:10:24.189937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.189849 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:24.190205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.190179 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:24.190301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.190278 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:24.583332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.582034 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:24.583332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:24.582093 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:25.023404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.023307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerStarted","Data":"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3"} Oct 29 22:10:25.023404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.023380 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerStarted","Data":"b3251c1d64a227e276cbc450be05285187d0239c5a0095747b70b5a5efee3bf8"} Oct 29 22:10:25.024987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.024960 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" event={"ID":"a294798d-a43a-40e7-b398-3e6132e976f9","Type":"ContainerStarted","Data":"7514d16a67d5574c70fb487dc230a85cc55e29a67bbeb5e3906ed5b0f83f2e5c"} Oct 29 22:10:25.025111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.025009 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" event={"ID":"a294798d-a43a-40e7-b398-3e6132e976f9","Type":"ContainerStarted","Data":"6f4654c4a07d95d3b16eea1099bc7430e2c87ca5f6c88f14467d7a304f8a1757"} Oct 29 22:10:25.025154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.025130 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:25.078274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.078225 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podStartSLOduration=3.078211057 podStartE2EDuration="3.078211057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:25.075788071 +0000 UTC m=+1683.924264960" watchObservedRunningTime="2026-10-29 22:10:25.078211057 +0000 UTC m=+1683.926687903" Oct 29 22:10:25.123138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.123076 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" podStartSLOduration=2.53848493 podStartE2EDuration="3.123058847s" totalImagesPullingTime="584.573917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:25.12253205 +0000 UTC m=+1683.971008919" watchObservedRunningTime="2026-10-29 22:10:25.123058847 +0000 UTC m=+1683.971535694" Oct 29 22:10:25.532389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.532356 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:25.563198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.563156 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:25.569183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.569156 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:25.569380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:25.569357 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:26.037953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:26.035452 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:26.037953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:26.036895 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:10:26.577960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:26.577574 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 22:10:26.577960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:26.577637 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 22:10:27.573743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:27.573708 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:27.574448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:27.574422 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:27.575165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:27.575145 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:10:31.040914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:31.040870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:10:46.632809 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:46.632763 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:10:48.938812 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:48.938777 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:10:48.979926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:48.970046 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.008000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.002147 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7721b3-serving-cert\"" Oct 29 22:10:49.130936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.122029 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.131502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.131466 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkc7\" (UniqueName: \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.138942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.131332 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:10:49.139403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.139368 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.139640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.139620 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.465924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.464379 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.465924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.464518 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.465924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.464661 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.465924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.464712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkc7\" (UniqueName: \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.485010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.477535 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.526968 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:49.500199 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-serving-cert: secret "maas-api-e2e-praxis-a-7721b3-serving-cert" not found Oct 29 22:10:49.526968 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:49.500308 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls podName:36b69b7b-2089-4d73-9e40-63f58f78cf78 nodeName:}" failed. No retries permitted until 2026-10-29 22:10:50.000275378 +0000 UTC m=+1708.848752216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls") pod "maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78") : secret "maas-api-e2e-praxis-a-7721b3-serving-cert" not found Oct 29 22:10:49.612507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.571657 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkc7\" (UniqueName: \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.621020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.620945 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:49.632036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.632006 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:10:49.638220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.638192 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.718331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.715192 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-665b09-openshift-default-dockercfg-fpgpt\"" Oct 29 22:10:49.888944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.868058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.902433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.902390 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.913194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.906157 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.925931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.913456 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.926290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.926252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.926640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.926619 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.926875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.926855 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.927054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.927034 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86nf\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.952311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.952273 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:49.953746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:49.953716 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:10:50.080295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.080259 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.102346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.102309 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.102647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.102621 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.102828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.102810 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.106867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.106825 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:50.107122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.107098 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.116599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.116561 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.116865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.116832 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86nf\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.117094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.117072 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.117389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.117364 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.117788 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:50.107617 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-serving-cert: secret "maas-api-e2e-praxis-a-7721b3-serving-cert" not found Oct 29 22:10:50.124057 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:10:50.124037 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls podName:36b69b7b-2089-4d73-9e40-63f58f78cf78 nodeName:}" failed. No retries permitted until 2026-10-29 22:10:51.123992413 +0000 UTC m=+1709.972469257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls") pod "maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78") : secret "maas-api-e2e-praxis-a-7721b3-serving-cert" not found Oct 29 22:10:50.124602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.106737 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.124756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.107508 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.124879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.123752 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.125032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.106471 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.125156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.080994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.153221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.153138 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.163592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.163520 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.198929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.194323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.284546 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.284476 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86nf\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\") pod \"e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.445268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.444938 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.599514 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.601052 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.601351 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": context deadline exceeded" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.602584 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.603570 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.608506 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:10:50.632932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.610017 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.634164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.633944 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.689106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:50.687321 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:10:51.143939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.125749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:51.324948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.297847 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7721b3-846ccc44-677c8\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:51.466971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.458700 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.664188 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.710896 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.711149 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.711558 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.711701 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.711854 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.712031 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.760057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.712145 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": context deadline exceeded" Oct 29 22:10:51.880776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:51.880499 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.245153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.245120 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:10:52.246039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246018 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:10:52.246222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246211 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:10:52.246352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246342 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:10:52.246486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246476 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:10:52.246604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246595 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:10:52.251715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.246709 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:10:52.254951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.252195 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:10:52.254951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.252249 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:10:52.285457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.285428 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:10:52.459931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.451088 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:10:52.459931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.453065 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:10:52.464006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.462434 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:10:52.465817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.464455 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:10:52.472749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.472660 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:10:52.474479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.474459 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:10:52.479844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.479788 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:10:52.504253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.486987 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:10:52.754801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.754734 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:10:52.757589 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:10:52.757555 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762bb4d0_b5d7_47ba_8671_b71f3d7684e6.slice/crio-a5ff3407454d6390648392d3f5471c1e261228d2d864ef1c3d3a2664cdaa834e WatchSource:0}: Error finding container a5ff3407454d6390648392d3f5471c1e261228d2d864ef1c3d3a2664cdaa834e: Status 404 returned error can't find the container with id a5ff3407454d6390648392d3f5471c1e261228d2d864ef1c3d3a2664cdaa834e Oct 29 22:10:52.774930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.774522 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:52.774930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.774616 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:52.774930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.774658 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:10:52.777980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.775321 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:10:52.784595 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:10:52.784562 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b69b7b_2089_4d73_9e40_63f58f78cf78.slice/crio-e4bc7577623ab5728d9ed298877c760b76bcd104da1468c1951702989e178052 WatchSource:0}: Error finding container e4bc7577623ab5728d9ed298877c760b76bcd104da1468c1951702989e178052: Status 404 returned error can't find the container with id e4bc7577623ab5728d9ed298877c760b76bcd104da1468c1951702989e178052 Oct 29 22:10:52.787024 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:52.787003 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:10:53.727021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.726977 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerStarted","Data":"1ca3f3ba67b7ee4a57566bb9ca99c7637f7b449cf7cb8f5a8a1628c114197a59"} Oct 29 22:10:53.727550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.727046 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerStarted","Data":"a5ff3407454d6390648392d3f5471c1e261228d2d864ef1c3d3a2664cdaa834e"} Oct 29 22:10:53.729161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.729130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" event={"ID":"36b69b7b-2089-4d73-9e40-63f58f78cf78","Type":"ContainerStarted","Data":"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e"} Oct 29 22:10:53.729314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.729185 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" event={"ID":"36b69b7b-2089-4d73-9e40-63f58f78cf78","Type":"ContainerStarted","Data":"e4bc7577623ab5728d9ed298877c760b76bcd104da1468c1951702989e178052"} Oct 29 22:10:53.729314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.729217 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:10:53.780984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.780932 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podStartSLOduration=4.78091874 podStartE2EDuration="4.78091874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:53.778738376 +0000 UTC m=+1712.627215244" watchObservedRunningTime="2026-10-29 22:10:53.78091874 +0000 UTC m=+1712.629395579" Oct 29 22:10:53.828417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:53.828364 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" podStartSLOduration=5.500085157 podStartE2EDuration="5.828350399s" totalImagesPullingTime="328.265242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:53.826569223 +0000 UTC m=+1712.675046071" watchObservedRunningTime="2026-10-29 22:10:53.828350399 +0000 UTC m=+1712.676827291" Oct 29 22:10:54.445672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:54.445638 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:54.451165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:54.451128 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:54.736855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:54.736816 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:54.737730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:54.737711 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:10:59.742007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:10:59.741978 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:11:04.344528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.344476 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:11:04.345062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.344716 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" podUID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" containerName="payload-processing" containerID="cri-o://1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee" gracePeriod=30 Oct 29 22:11:04.383467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.383420 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:11:04.383889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.383662 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" podUID="639993b7-932e-4bbd-b997-990ceda36487" containerName="payload-pre-processing" containerID="cri-o://da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55" gracePeriod=30 Oct 29 22:11:04.712679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.712651 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:11:04.770926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.770160 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.880508 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\") pod \"639993b7-932e-4bbd-b997-990ceda36487\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.880579 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\") pod \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.880626 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\" (UniqueName: \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\") pod \"639993b7-932e-4bbd-b997-990ceda36487\" (UID: \"639993b7-932e-4bbd-b997-990ceda36487\") " Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.880826 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\" (UniqueName: \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\") pod \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\" (UID: \"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a\") " Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.883373 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume" pod "b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" (UID: "b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:04.884975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.883814 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume" pod "639993b7-932e-4bbd-b997-990ceda36487" (UID: "639993b7-932e-4bbd-b997-990ceda36487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:04.930952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.916428 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8" pod "b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" (UID: "b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a"). InnerVolumeSpecName "kube-api-access-jm2p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:04.933472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.933408 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s" pod "639993b7-932e-4bbd-b997-990ceda36487" (UID: "639993b7-932e-4bbd-b997-990ceda36487"). InnerVolumeSpecName "kube-api-access-xrk5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:04.984946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.981372 4376 generic.go:356] "Generic (PLEG): container finished" podID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" containerID="1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee" exitCode=0 Oct 29 22:11:04.984946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.981593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" event={"ID":"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a","Type":"ContainerDied","Data":"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee"} Oct 29 22:11:04.984946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.981665 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" event={"ID":"b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a","Type":"ContainerDied","Data":"2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8"} Oct 29 22:11:04.984946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.981706 4376 scope.go:122] "RemoveContainer" containerID="1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee" Oct 29 22:11:04.984946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:04.981935 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5" Oct 29 22:11:05.046943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.015079 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2p8\" (UniqueName: \"kubernetes.io/projected/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-kube-api-access-jm2p8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:05.046943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.015117 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639993b7-932e-4bbd-b997-990ceda36487-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:05.046943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.015187 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:05.046943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.015203 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrk5s\" (UniqueName: \"kubernetes.io/projected/639993b7-932e-4bbd-b997-990ceda36487-kube-api-access-xrk5s\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:05.099547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.099518 4376 generic.go:356] "Generic (PLEG): container finished" podID="639993b7-932e-4bbd-b997-990ceda36487" containerID="da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55" exitCode=0 Oct 29 22:11:05.099660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.099646 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" event={"ID":"639993b7-932e-4bbd-b997-990ceda36487","Type":"ContainerDied","Data":"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55"} Oct 29 22:11:05.099736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.099685 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" event={"ID":"639993b7-932e-4bbd-b997-990ceda36487","Type":"ContainerDied","Data":"1ea85f25e95facdf551221781d65033df68b2aaec871c5c68195c1290ec00a14"} Oct 29 22:11:05.100125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.100102 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw" Oct 29 22:11:05.117569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.117538 4376 scope.go:122] "RemoveContainer" containerID="1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee" Oct 29 22:11:05.118637 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:11:05.118609 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee\": container with ID starting with 1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee not found: ID does not exist" containerID="1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee" Oct 29 22:11:05.118738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.118651 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee"} err="failed to get container status \"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee\": rpc error: code = NotFound desc = could not find container \"1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee\": container with ID starting with 1b80650c595de252f0b1125d8b513fa4759d90ae082d8034886830f9733ce1ee not found: ID does not exist" Oct 29 22:11:05.118738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.118676 4376 scope.go:122] "RemoveContainer" containerID="da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55" Oct 29 22:11:05.130144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.130121 4376 scope.go:122] "RemoveContainer" containerID="da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55" Oct 29 22:11:05.130466 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:11:05.130441 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55\": container with ID starting with da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55 not found: ID does not exist" containerID="da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55" Oct 29 22:11:05.130549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.130478 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55"} err="failed to get container status \"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55\": rpc error: code = NotFound desc = could not find container \"da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55\": container with ID starting with da79d095c33b81832c3c72daec231658c20f048b07da7095840e974fde582f55 not found: ID does not exist" Oct 29 22:11:05.146932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.145838 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:11:05.150502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.150472 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fe45b-559999b564-dmzg5"] Oct 29 22:11:05.180594 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:11:05.180549 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a33cc_0d5a_4ff8_a118_3d8d6a71d82a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639993b7_932e_4bbd_b997_990ceda36487.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a33cc_0d5a_4ff8_a118_3d8d6a71d82a.slice/crio-2f1a6cc2434277d0ffdd3e10ec28feae9426cd28291fd16e695ae2f2343736d8\": RecentStats: unable to find data in memory cache]" Oct 29 22:11:05.190246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.190216 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:11:05.194825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.194797 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-56644bf8c9-55rqw"] Oct 29 22:11:05.275007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.274970 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639993b7-932e-4bbd-b997-990ceda36487" path="/var/lib/kubelet/pods/639993b7-932e-4bbd-b997-990ceda36487/volumes" Oct 29 22:11:05.275433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:05.275418 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" path="/var/lib/kubelet/pods/b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a/volumes" Oct 29 22:11:10.714891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:10.714846 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:11:16.434561 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:11:16.434517 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:11:20.718926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.718074 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:11:20.719613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719464 4376 cpu_manager.go:414] "containerMap: removing container" podUID="639993b7-932e-4bbd-b997-990ceda36487" containerName="payload-pre-processing" Oct 29 22:11:20.719613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719500 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639993b7-932e-4bbd-b997-990ceda36487" containerName="payload-pre-processing" Oct 29 22:11:20.719613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719529 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" containerName="payload-processing" Oct 29 22:11:20.719613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719538 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" containerName="payload-processing" Oct 29 22:11:20.719847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719799 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="639993b7-932e-4bbd-b997-990ceda36487" containerName="payload-pre-processing" Oct 29 22:11:20.719847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.719818 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68a33cc-0d5a-4ff8-a118-3d8d6a71d82a" containerName="payload-processing" Oct 29 22:11:20.749280 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.749245 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.759240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.759196 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:11:20.818985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.816924 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-665b09-serving-cert\"" Oct 29 22:11:20.850166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.850129 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4db\" (UniqueName: \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.850349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.850194 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.850349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.850239 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.850349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.850311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.962926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.956980 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4db\" (UniqueName: \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.962926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.957078 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.962926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.957146 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.962926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.957257 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.962926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.958349 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:20.988976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:20.988291 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:21.004930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:21.001701 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4db\" (UniqueName: \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:21.029171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:21.015461 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\") pod \"maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:21.132684 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:21.132651 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:21.447163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:21.447135 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:11:21.448068 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:11:21.448038 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef07284b_1a12_49e3_a536_9c7692005c9b.slice/crio-01dcfc1be3ff67fcfbeaa0cb7cbb7a65a2eb69e5fec3f0920b58fc6b2e8492a0 WatchSource:0}: Error finding container 01dcfc1be3ff67fcfbeaa0cb7cbb7a65a2eb69e5fec3f0920b58fc6b2e8492a0: Status 404 returned error can't find the container with id 01dcfc1be3ff67fcfbeaa0cb7cbb7a65a2eb69e5fec3f0920b58fc6b2e8492a0 Oct 29 22:11:22.360969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:22.360930 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" event={"ID":"ef07284b-1a12-49e3-a536-9c7692005c9b","Type":"ContainerStarted","Data":"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4"} Oct 29 22:11:22.361665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:22.361636 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" event={"ID":"ef07284b-1a12-49e3-a536-9c7692005c9b","Type":"ContainerStarted","Data":"01dcfc1be3ff67fcfbeaa0cb7cbb7a65a2eb69e5fec3f0920b58fc6b2e8492a0"} Oct 29 22:11:22.362062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:22.362038 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:22.414097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:22.414033 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" podStartSLOduration=2.031486813 podStartE2EDuration="2.414018684s" totalImagesPullingTime="382.531871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:22.409826567 +0000 UTC m=+1741.258303424" watchObservedRunningTime="2026-10-29 22:11:22.414018684 +0000 UTC m=+1741.262495530" Oct 29 22:11:25.757237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:25.757146 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" containerID="cri-o://0fb4fc6aa6d222ddc1b811f8a7c51970ddf6000859a37f26ec21ca668b41a633" gracePeriod=45 Oct 29 22:11:26.107169 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:26.106649 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": EOF" Oct 29 22:11:26.469266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:26.467978 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": dial tcp 10.134.0.132:8000: connect: connection refused" Oct 29 22:11:26.469266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:26.468127 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:11:26.875669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:26.875632 4376 generic.go:356] "Generic (PLEG): container finished" podID="acd40135-543f-4b81-9d20-964f0c60a12f" containerID="0fb4fc6aa6d222ddc1b811f8a7c51970ddf6000859a37f26ec21ca668b41a633" exitCode=0 Oct 29 22:11:26.908045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:26.895481 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" event={"ID":"acd40135-543f-4b81-9d20-964f0c60a12f","Type":"ContainerDied","Data":"0fb4fc6aa6d222ddc1b811f8a7c51970ddf6000859a37f26ec21ca668b41a633"} Oct 29 22:11:27.354970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.330672 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.477530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.475469 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.477530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.476257 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": context deadline exceeded" Oct 29 22:11:27.477530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.476882 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:11:27.477530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.477069 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:11:27.477530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.477218 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 29 22:11:27.497920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.483153 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:11:27.602929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.594094 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.770238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:27.754274 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.443154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.442503 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:11:28.444140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.444108 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.482237 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.484970 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.485155 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": context deadline exceeded" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.485348 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.485713 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": context deadline exceeded" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.485825 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 29 22:11:28.631222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.504508 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 29 22:11:28.760927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.665042 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:11:28.760927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.698079 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.824439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:28.820264 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 29 22:11:29.054157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.054125 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:11:29.092310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.086242 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:11:29.169280 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.169251 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:11:29.169587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.169566 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:11:29.169844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.169829 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:11:29.172809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.172791 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:11:29.173036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.173012 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:11:29.173625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.173610 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:11:29.173833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.173819 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:11:29.187611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.187583 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:11:29.188332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.188311 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:11:29.292486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.289946 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:11:29.364453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.364358 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.364848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.364817 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.365121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.365098 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\" (UniqueName: \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.365331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.365314 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.365498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.365484 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\" (UniqueName: \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.365640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.365627 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\") pod \"acd40135-543f-4b81-9d20-964f0c60a12f\" (UID: \"acd40135-543f-4b81-9d20-964f0c60a12f\") " Oct 29 22:11:29.369023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.368999 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:29.372233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.372177 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:29.373438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.373410 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:29.374035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.374014 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:29.378775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.375360 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "kube-api-access-bmngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:29.401686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.401630 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs" pod "acd40135-543f-4b81-9d20-964f0c60a12f" (UID: "acd40135-543f-4b81-9d20-964f0c60a12f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:29.429649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.429625 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:11:29.429790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.429678 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:11:29.429790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.429714 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:11:29.429790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.429746 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:11:29.468253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468219 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.468253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468248 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd40135-543f-4b81-9d20-964f0c60a12f-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.468253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468257 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.468974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468269 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.468974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468280 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd40135-543f-4b81-9d20-964f0c60a12f-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.468974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.468289 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmngk\" (UniqueName: \"kubernetes.io/projected/acd40135-543f-4b81-9d20-964f0c60a12f-kube-api-access-bmngk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:29.685442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:29.685350 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:30.117736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.117703 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" Oct 29 22:11:30.117941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.117740 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw" event={"ID":"acd40135-543f-4b81-9d20-964f0c60a12f","Type":"ContainerDied","Data":"df0730f44e7ed7b1b8a0103c2e841b111b0b39548c1bc8c3cfa1c540bd81c4d2"} Oct 29 22:11:30.117941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.117795 4376 scope.go:122] "RemoveContainer" containerID="0fb4fc6aa6d222ddc1b811f8a7c51970ddf6000859a37f26ec21ca668b41a633" Oct 29 22:11:30.200140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.196333 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:11:30.203192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.203164 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3fe45b/ipp-route-3fe45b-kserve-587d45885-wfwjw"] Oct 29 22:11:30.415973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.413011 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:11:30.415973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.414145 4376 cpu_manager.go:414] "containerMap: removing container" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" Oct 29 22:11:30.415973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.414168 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" Oct 29 22:11:30.415973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.414405 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" containerName="main" Oct 29 22:11:30.454131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.443826 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:30.820723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.808315 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4c5549-dockercfg-lsggv\"" Oct 29 22:11:30.820723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.808438 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4c5549\"" Oct 29 22:11:30.920471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.920423 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgw4g\" (UniqueName: \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:30.920745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.920723 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:30.930070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:30.930040 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:11:31.022094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.022054 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgw4g\" (UniqueName: \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:31.022277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.022208 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:31.022994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.022975 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:31.060930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.060738 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgw4g\" (UniqueName: \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\") pod \"payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:31.269670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.269637 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:31.275720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.275683 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd40135-543f-4b81-9d20-964f0c60a12f" path="/var/lib/kubelet/pods/acd40135-543f-4b81-9d20-964f0c60a12f/volumes" Oct 29 22:11:31.703162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:31.702719 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:11:32.159807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:32.159773 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" event={"ID":"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca","Type":"ContainerStarted","Data":"e2c1678c8d2613c3eedb17399fd160e72ad2407cd10eafc87ec9d51e633c3d3f"} Oct 29 22:11:32.160283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:32.159834 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" event={"ID":"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca","Type":"ContainerStarted","Data":"ec3541cb47cfc2afe1db3e6f23b38e8a267979c4c1a8ccfd5201416ff3807d08"} Oct 29 22:11:32.160283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:32.159877 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:32.207604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:32.207545 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" podStartSLOduration=2.207529624 podStartE2EDuration="2.207529624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:32.204019138 +0000 UTC m=+1751.052495984" watchObservedRunningTime="2026-10-29 22:11:32.207529624 +0000 UTC m=+1751.056006470" Oct 29 22:11:38.206958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.206882 4376 generic.go:356] "Generic (PLEG): container finished" podID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerID="138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3" exitCode=0 Oct 29 22:11:38.207419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.206956 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerDied","Data":"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3"} Oct 29 22:11:38.207500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.207485 4376 scope.go:122] "RemoveContainer" containerID="138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3" Oct 29 22:11:38.209751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.209688 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:11:38.209893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.209809 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:11:38.209893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:38.209865 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:11:39.216462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:39.216426 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerStarted","Data":"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd"} Oct 29 22:11:39.532244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:39.532157 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:11:39.556363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:39.556324 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:39.556697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:39.556653 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:40.532499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:40.532460 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:40.532990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:40.532534 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:41.058097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:41.058015 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:11:41.533200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:41.533163 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:41.533862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:41.533819 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:42.532147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:42.532111 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:42.532357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:42.532182 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:43.173020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:43.172983 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:11:43.532608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:43.532568 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:43.532803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:43.532634 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:44.532335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:44.532298 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:11:44.532754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:44.532354 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:11:45.554003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:45.553953 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:11:45.559859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:45.559821 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:11:46.436546 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:11:46.436505 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:11:51.617979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.617932 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:11:51.621720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.621695 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.646943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.646872 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-05687d-serving-cert\"" Oct 29 22:11:51.647132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.647097 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:11:51.687953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.687898 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.688155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.687996 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.688155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.688021 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm88p\" (UniqueName: \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.688245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.688212 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.821924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.821550 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.821924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.821690 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.821924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.821723 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm88p\" (UniqueName: \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.821924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.821814 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.826957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.822988 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.877281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.874994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.892926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.877485 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.905015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.895253 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm88p\" (UniqueName: \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\") pod \"maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:51.943779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:51.943353 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:52.264525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:52.264499 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:11:52.265883 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:11:52.265855 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2b9788_57fe_4ce5_8386_80ea9d248a61.slice/crio-660c6e8d1e55bf85aa229251febeb1597499eaadc0f2016292912bddffee341f WatchSource:0}: Error finding container 660c6e8d1e55bf85aa229251febeb1597499eaadc0f2016292912bddffee341f: Status 404 returned error can't find the container with id 660c6e8d1e55bf85aa229251febeb1597499eaadc0f2016292912bddffee341f Oct 29 22:11:52.509915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:52.509867 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" event={"ID":"0e2b9788-57fe-4ce5-8386-80ea9d248a61","Type":"ContainerStarted","Data":"660c6e8d1e55bf85aa229251febeb1597499eaadc0f2016292912bddffee341f"} Oct 29 22:11:53.518673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:53.518640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" event={"ID":"0e2b9788-57fe-4ce5-8386-80ea9d248a61","Type":"ContainerStarted","Data":"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2"} Oct 29 22:11:53.519078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:53.518844 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:11:53.568574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:53.568517 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" podStartSLOduration=2.25938769 podStartE2EDuration="2.568502862s" totalImagesPullingTime="309.115172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:53.564254491 +0000 UTC m=+1772.412731338" watchObservedRunningTime="2026-10-29 22:11:53.568502862 +0000 UTC m=+1772.416979707" Oct 29 22:11:56.398216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:56.398173 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:11:56.398806 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:56.398460 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" containerID="cri-o://1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831" gracePeriod=30 Oct 29 22:11:59.606596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:11:59.606559 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:12:01.690937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.690898 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:12:01.692102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.692072 4376 generic.go:356] "Generic (PLEG): container finished" podID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerID="1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831" exitCode=0 Oct 29 22:12:01.692213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.692101 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerDied","Data":"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831"} Oct 29 22:12:01.692213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.692169 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" event={"ID":"bf5f61f2-62d1-47fa-b990-234e45df1f83","Type":"ContainerDied","Data":"d8584dbaab8b47d37ac134ab05ffc5d566b606e0009f25f510a665ef542fba00"} Oct 29 22:12:01.692213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.692198 4376 scope.go:122] "RemoveContainer" containerID="1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831" Oct 29 22:12:01.705570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.704064 4376 scope.go:122] "RemoveContainer" containerID="e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c" Oct 29 22:12:01.714783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.714765 4376 scope.go:122] "RemoveContainer" containerID="1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831" Oct 29 22:12:01.715077 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:01.715055 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831\": container with ID starting with 1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831 not found: ID does not exist" containerID="1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831" Oct 29 22:12:01.715147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.715086 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831"} err="failed to get container status \"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831\": rpc error: code = NotFound desc = could not find container \"1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831\": container with ID starting with 1ad3ded630374c537bf9e19634c5bdcb70aa159c10ca0290ffd06bec9b1fc831 not found: ID does not exist" Oct 29 22:12:01.715147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.715105 4376 scope.go:122] "RemoveContainer" containerID="e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c" Oct 29 22:12:01.715332 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:01.715311 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c\": container with ID starting with e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c not found: ID does not exist" containerID="e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c" Oct 29 22:12:01.715374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.715338 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c"} err="failed to get container status \"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c\": rpc error: code = NotFound desc = could not find container \"e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c\": container with ID starting with e3b4d79e7add2ed74bce8d488916dd387536e1d85cb7131f40dceae19e55a39c not found: ID does not exist" Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826508 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826632 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826676 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826815 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826858 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826924 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.826976 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.827075 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.828241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.827189 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\") pod \"bf5f61f2-62d1-47fa-b990-234e45df1f83\" (UID: \"bf5f61f2-62d1-47fa-b990-234e45df1f83\") " Oct 29 22:12:01.829258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.828936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.881962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.873263 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.881962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.880193 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.881962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.880657 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:01.897774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.897648 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.947002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.938675 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5f61f2-62d1-47fa-b990-234e45df1f83-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.947002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.938790 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.947002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.938813 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.947002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.938826 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.947002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.938888 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.994026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.993883 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:01.997127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:01.997073 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:02.045209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.045168 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "kube-api-access-cpnjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:02.058131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.045322 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token" pod "bf5f61f2-62d1-47fa-b990-234e45df1f83" (UID: "bf5f61f2-62d1-47fa-b990-234e45df1f83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:02.058332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.048691 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:02.058481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.058466 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:02.160874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.160835 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnjl\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-kube-api-access-cpnjl\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:02.160874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.160870 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5f61f2-62d1-47fa-b990-234e45df1f83-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:02.702576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.702545 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88" Oct 29 22:12:02.777844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.777799 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:12:02.791305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:02.791269 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-45n88"] Oct 29 22:12:03.387962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:03.383634 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" path="/var/lib/kubelet/pods/bf5f61f2-62d1-47fa-b990-234e45df1f83/volumes" Oct 29 22:12:04.177593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:04.177524 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 29 22:12:08.466978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:08.464897 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:12:08.467953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:08.467898 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" podUID="a294798d-a43a-40e7-b398-3e6132e976f9" containerName="maas-api" containerID="cri-o://7514d16a67d5574c70fb487dc230a85cc55e29a67bbeb5e3906ed5b0f83f2e5c" gracePeriod=30 Oct 29 22:12:08.989114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:08.972972 4376 generic.go:356] "Generic (PLEG): container finished" podID="a294798d-a43a-40e7-b398-3e6132e976f9" containerID="7514d16a67d5574c70fb487dc230a85cc55e29a67bbeb5e3906ed5b0f83f2e5c" exitCode=0 Oct 29 22:12:08.989114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:08.973331 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" event={"ID":"a294798d-a43a-40e7-b398-3e6132e976f9","Type":"ContainerDied","Data":"7514d16a67d5574c70fb487dc230a85cc55e29a67bbeb5e3906ed5b0f83f2e5c"} Oct 29 22:12:09.463662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.451032 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:12:09.565311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.565262 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") pod \"a294798d-a43a-40e7-b398-3e6132e976f9\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " Oct 29 22:12:09.566281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.566250 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\") pod \"a294798d-a43a-40e7-b398-3e6132e976f9\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " Oct 29 22:12:09.567047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.567026 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\") pod \"a294798d-a43a-40e7-b398-3e6132e976f9\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " Oct 29 22:12:09.567964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.567932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\" (UniqueName: \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\") pod \"a294798d-a43a-40e7-b398-3e6132e976f9\" (UID: \"a294798d-a43a-40e7-b398-3e6132e976f9\") " Oct 29 22:12:09.568960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.566940 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca" pod "a294798d-a43a-40e7-b398-3e6132e976f9" (UID: "a294798d-a43a-40e7-b398-3e6132e976f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:09.628391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.623120 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w" pod "a294798d-a43a-40e7-b398-3e6132e976f9" (UID: "a294798d-a43a-40e7-b398-3e6132e976f9"). InnerVolumeSpecName "kube-api-access-hcw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:09.651416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.651380 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a294798d-a43a-40e7-b398-3e6132e976f9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.656173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.656143 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcw9w\" (UniqueName: \"kubernetes.io/projected/a294798d-a43a-40e7-b398-3e6132e976f9-kube-api-access-hcw9w\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.656879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.656851 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls" pod "a294798d-a43a-40e7-b398-3e6132e976f9" (UID: "a294798d-a43a-40e7-b398-3e6132e976f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:09.771418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.771336 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs" pod "a294798d-a43a-40e7-b398-3e6132e976f9" (UID: "a294798d-a43a-40e7-b398-3e6132e976f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:09.828933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.827170 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.828933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:09.827209 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a294798d-a43a-40e7-b398-3e6132e976f9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.009378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:10.009353 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" Oct 29 22:12:10.009649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:10.009382 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv" event={"ID":"a294798d-a43a-40e7-b398-3e6132e976f9","Type":"ContainerDied","Data":"6f4654c4a07d95d3b16eea1099bc7430e2c87ca5f6c88f14467d7a304f8a1757"} Oct 29 22:12:10.009846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:10.009832 4376 scope.go:122] "RemoveContainer" containerID="7514d16a67d5574c70fb487dc230a85cc55e29a67bbeb5e3906ed5b0f83f2e5c" Oct 29 22:12:10.073808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:10.073775 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:12:10.199924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:10.198329 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80bd36-7895895fd7-smjrv"] Oct 29 22:12:11.273486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:11.273451 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a294798d-a43a-40e7-b398-3e6132e976f9" path="/var/lib/kubelet/pods/a294798d-a43a-40e7-b398-3e6132e976f9/volumes" Oct 29 22:12:13.527973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.527935 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:12:13.528651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528635 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.528695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528653 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.528695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528669 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a294798d-a43a-40e7-b398-3e6132e976f9" containerName="maas-api" Oct 29 22:12:13.528695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528674 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a294798d-a43a-40e7-b398-3e6132e976f9" containerName="maas-api" Oct 29 22:12:13.528695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528692 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.528819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528697 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.528819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528811 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.528884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528820 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a294798d-a43a-40e7-b398-3e6132e976f9" containerName="maas-api" Oct 29 22:12:13.528884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.528829 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5f61f2-62d1-47fa-b990-234e45df1f83" containerName="istio-proxy" Oct 29 22:12:13.534685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.534657 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.558441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.558408 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:12:13.563231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.563201 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-99e420-serving-cert\"" Oct 29 22:12:13.585100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.585065 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7tcb\" (UniqueName: \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.585297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.585262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.585533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.585509 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.585617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.585552 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.686883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.686846 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.686883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.686887 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.687135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.687033 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7tcb\" (UniqueName: \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.687135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.687108 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.687796 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.687764 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.689733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.689704 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.689857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.689704 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.713480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.713451 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7tcb\" (UniqueName: \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\") pod \"maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:13.857632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:13.856623 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:14.232209 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:12:14.232173 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc313ec_8b15_48da_9467_c70f2282594f.slice/crio-5f161452bc73df21bd41b04a5817fd1d59b9bcdfd0fa8ece6b8aa0333970deca WatchSource:0}: Error finding container 5f161452bc73df21bd41b04a5817fd1d59b9bcdfd0fa8ece6b8aa0333970deca: Status 404 returned error can't find the container with id 5f161452bc73df21bd41b04a5817fd1d59b9bcdfd0fa8ece6b8aa0333970deca Oct 29 22:12:14.232421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:14.232401 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.066938 4376 generic.go:356] "Generic (PLEG): container finished" podID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerID="1ca3f3ba67b7ee4a57566bb9ca99c7637f7b449cf7cb8f5a8a1628c114197a59" exitCode=0 Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.067241 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerDied","Data":"1ca3f3ba67b7ee4a57566bb9ca99c7637f7b449cf7cb8f5a8a1628c114197a59"} Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.067940 4376 scope.go:122] "RemoveContainer" containerID="1ca3f3ba67b7ee4a57566bb9ca99c7637f7b449cf7cb8f5a8a1628c114197a59" Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.072713 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.072819 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:15.075512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.072869 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:15.079436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.079402 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" event={"ID":"4fc313ec-8b15-48da-9467-c70f2282594f","Type":"ContainerStarted","Data":"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45"} Oct 29 22:12:15.079603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.079466 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" event={"ID":"4fc313ec-8b15-48da-9467-c70f2282594f","Type":"ContainerStarted","Data":"5f161452bc73df21bd41b04a5817fd1d59b9bcdfd0fa8ece6b8aa0333970deca"} Oct 29 22:12:15.079603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.079572 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:15.110480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.110446 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:12:15.110704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.110678 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" podUID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" containerName="maas-api" containerID="cri-o://bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c" gracePeriod=30 Oct 29 22:12:15.238161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.238113 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" podStartSLOduration=1.635024045 podStartE2EDuration="2.238096646s" totalImagesPullingTime="603.072601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:15.236445907 +0000 UTC m=+1794.084922776" watchObservedRunningTime="2026-10-29 22:12:15.238096646 +0000 UTC m=+1794.086573493" Oct 29 22:12:15.454672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.454642 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:12:15.489353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.489317 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\") pod \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " Oct 29 22:12:15.489624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.489391 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\") pod \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " Oct 29 22:12:15.489624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.489488 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\") pod \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " Oct 29 22:12:15.489794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.489769 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\" (UniqueName: \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\") pod \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\" (UID: \"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4\") " Oct 29 22:12:15.490150 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.490122 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca" pod "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" (UID: "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:15.490604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.490580 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.492211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.492177 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls" pod "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" (UID: "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:15.492295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.492248 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g" pod "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" (UID: "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4"). InnerVolumeSpecName "kube-api-access-qrd2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:15.492456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.492440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs" pod "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" (UID: "26b3bdef-d3ca-4458-bfe3-b708e1a4eab4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:15.519422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.519387 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:12:15.592318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.592271 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.592318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.592315 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.592584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:15.592331 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrd2g\" (UniqueName: \"kubernetes.io/projected/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4-kube-api-access-qrd2g\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:16.160671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.157284 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerStarted","Data":"de975296a9fb122fad97e62ef21489b3c25bf590b60667dfc1a9bccebc2d7701"} Oct 29 22:12:16.170990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.163972 4376 generic.go:356] "Generic (PLEG): container finished" podID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" containerID="bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c" exitCode=0 Oct 29 22:12:16.170990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.165251 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" Oct 29 22:12:16.170990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.168953 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" event={"ID":"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4","Type":"ContainerDied","Data":"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c"} Oct 29 22:12:16.170990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.169020 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp" event={"ID":"26b3bdef-d3ca-4458-bfe3-b708e1a4eab4","Type":"ContainerDied","Data":"9984707238c3ba711cf926b702d59e1340a6e545bce953dfb7ba88e1fdaeba48"} Oct 29 22:12:16.170990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.169052 4376 scope.go:122] "RemoveContainer" containerID="bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c" Oct 29 22:12:16.190009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.189984 4376 scope.go:122] "RemoveContainer" containerID="bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c" Oct 29 22:12:16.190341 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:16.190316 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c\": container with ID starting with bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c not found: ID does not exist" containerID="bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c" Oct 29 22:12:16.190391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.190353 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c"} err="failed to get container status \"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c\": rpc error: code = NotFound desc = could not find container \"bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c\": container with ID starting with bdf1ebe7297987410e07d1ca5d28dd82b102c491f6829b11a6a5e99c212b0a4c not found: ID does not exist" Oct 29 22:12:16.261232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.261176 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:12:16.264986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.264953 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a20be1-9768d86d6-twkvp"] Oct 29 22:12:16.455990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.452324 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:16.455990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.452456 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:16.455990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:16.452503 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:16.461455 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:16.461418 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:12:17.381941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:17.371674 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" path="/var/lib/kubelet/pods/26b3bdef-d3ca-4458-bfe3-b708e1a4eab4/volumes" Oct 29 22:12:17.450093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:17.450059 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:17.450399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:17.450357 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:18.455926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:18.455303 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:18.455926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:18.455370 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:19.454928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:19.451880 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:19.454928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:19.452092 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:20.450931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:20.447993 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:20.450931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:20.448059 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:21.170877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:21.170843 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:12:21.458979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:21.446954 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 22:12:21.458979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:21.453385 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 22:12:22.512096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:22.512059 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:22.512655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:22.512583 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:22.514459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:22.514433 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:23.847107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:23.847060 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:12:23.847995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:23.847962 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" containerID="cri-o://f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8" gracePeriod=30 Oct 29 22:12:29.324380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.324348 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:12:29.369376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369621 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369679 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:29.369727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369695 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369771 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369852 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.369977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.369930 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:29.370133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370107 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.370331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370304 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:29.370437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370320 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370556 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370561 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\") pod \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\" (UID: \"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6\") " Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.370750 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.371459 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.371483 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.371499 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.371514 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.371554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.371528 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.372384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.372357 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:29.372792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.372769 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:29.372873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.372852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:29.372964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.372943 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd" pod "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" (UID: "3f2097b8-6e8f-4e66-be1a-0aee6587d7d6"). InnerVolumeSpecName "kube-api-access-v96sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:29.473374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.473329 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.473374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.473372 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96sd\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-kube-api-access-v96sd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.473603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.473388 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.473603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.473401 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:29.494158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.494118 4376 generic.go:356] "Generic (PLEG): container finished" podID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerID="f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8" exitCode=0 Oct 29 22:12:29.494347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.494169 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" Oct 29 22:12:29.494347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.494186 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerDied","Data":"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8"} Oct 29 22:12:29.494347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.494246 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9" event={"ID":"3f2097b8-6e8f-4e66-be1a-0aee6587d7d6","Type":"ContainerDied","Data":"f2dc0e1ae4b50c807470003f5edf62be71aac85e731f45135770504f17451b3f"} Oct 29 22:12:29.494347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.494279 4376 scope.go:122] "RemoveContainer" containerID="f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8" Oct 29 22:12:29.506059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.506037 4376 scope.go:122] "RemoveContainer" containerID="fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6" Oct 29 22:12:29.516355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.516335 4376 scope.go:122] "RemoveContainer" containerID="f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8" Oct 29 22:12:29.516630 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:29.516611 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8\": container with ID starting with f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8 not found: ID does not exist" containerID="f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8" Oct 29 22:12:29.516674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.516642 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8"} err="failed to get container status \"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8\": rpc error: code = NotFound desc = could not find container \"f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8\": container with ID starting with f731f425f2157371aa9070f7f6b3a0bec73da0ced93c60702891eac1df3940b8 not found: ID does not exist" Oct 29 22:12:29.516674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.516660 4376 scope.go:122] "RemoveContainer" containerID="fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6" Oct 29 22:12:29.517004 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:29.516920 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6\": container with ID starting with fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6 not found: ID does not exist" containerID="fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6" Oct 29 22:12:29.517004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.516946 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6"} err="failed to get container status \"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6\": rpc error: code = NotFound desc = could not find container \"fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6\": container with ID starting with fc570db7127d23c58bc203fe279992a2a4cb46b5fd0f8ba24d86e48e58791ca6 not found: ID does not exist" Oct 29 22:12:29.630645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.630007 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:12:29.699984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:29.668097 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a20be1-openshift-default-fd4c645c5-g6hf9"] Oct 29 22:12:30.546131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:30.546088 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" containerID="cri-o://eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8" gracePeriod=45 Oct 29 22:12:30.837050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:30.837025 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:12:31.012799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.012769 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\" (UniqueName: \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.012979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.012814 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.012979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.012857 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.012979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.012940 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\" (UniqueName: \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.013180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.013110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.013180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.013137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:31.013258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.013184 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:31.013258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.013221 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\") pod \"46d3b6ce-d0f1-4313-8e18-907fc389745a\" (UID: \"46d3b6ce-d0f1-4313-8e18-907fc389745a\") " Oct 29 22:12:31.013427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.013402 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:31.014032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.014000 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.014032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.014023 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.014032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.014032 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.015348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.015326 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:31.015467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.015328 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:31.015562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.015536 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99" pod "46d3b6ce-d0f1-4313-8e18-907fc389745a" (UID: "46d3b6ce-d0f1-4313-8e18-907fc389745a"). InnerVolumeSpecName "kube-api-access-5pg99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:31.117728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.117619 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d3b6ce-d0f1-4313-8e18-907fc389745a-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.117987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.117968 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d3b6ce-d0f1-4313-8e18-907fc389745a-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.118125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.118107 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg99\" (UniqueName: \"kubernetes.io/projected/46d3b6ce-d0f1-4313-8e18-907fc389745a-kube-api-access-5pg99\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.274450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.274414 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" path="/var/lib/kubelet/pods/3f2097b8-6e8f-4e66-be1a-0aee6587d7d6/volumes" Oct 29 22:12:31.544992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.544951 4376 generic.go:356] "Generic (PLEG): container finished" podID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerID="eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8" exitCode=0 Oct 29 22:12:31.545362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.545332 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" event={"ID":"46d3b6ce-d0f1-4313-8e18-907fc389745a","Type":"ContainerDied","Data":"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8"} Oct 29 22:12:31.545569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.545547 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" event={"ID":"46d3b6ce-d0f1-4313-8e18-907fc389745a","Type":"ContainerDied","Data":"0f08350e757d5d5544ae8de516c072ecc9c3c5a46cfbb259e449ae65e4ce21ea"} Oct 29 22:12:31.545829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.545810 4376 scope.go:122] "RemoveContainer" containerID="eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8" Oct 29 22:12:31.546112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.546095 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm" Oct 29 22:12:31.598145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.594124 4376 scope.go:122] "RemoveContainer" containerID="eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8" Oct 29 22:12:31.598820 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:31.598780 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8\": container with ID starting with eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8 not found: ID does not exist" containerID="eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8" Oct 29 22:12:31.600533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.600499 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8"} err="failed to get container status \"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8\": rpc error: code = NotFound desc = could not find container \"eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8\": container with ID starting with eed5e929f562d7796f8c05b1eb59020a565f4e81a3982917c79b1d0675f6fdc8 not found: ID does not exist" Oct 29 22:12:31.609495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.609465 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:12:31.615276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:31.615250 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-796d71/praxis-route-796d71-kserve-5b7dc5f65c-9bzkm"] Oct 29 22:12:32.982023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.981980 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:12:32.982713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982697 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:32.982764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982714 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:32.982764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982730 4376 cpu_manager.go:414] "containerMap: removing container" podUID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" containerName="maas-api" Oct 29 22:12:32.982764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982736 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" containerName="maas-api" Oct 29 22:12:32.982856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982774 4376 cpu_manager.go:414] "containerMap: removing container" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" Oct 29 22:12:32.982856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982779 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" Oct 29 22:12:32.982929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982878 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:32.982929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982887 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b3bdef-d3ca-4458-bfe3-b708e1a4eab4" containerName="maas-api" Oct 29 22:12:32.982929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.982898 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" containerName="main" Oct 29 22:12:32.986461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:32.986442 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.008567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.008534 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:12:33.017675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.017647 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-76c1b7\"" Oct 29 22:12:33.017830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.017732 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-76c1b7-dockercfg-zn8w9\"" Oct 29 22:12:33.094930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.053097 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7lr\" (UniqueName: \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.201931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.166970 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.296372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.296271 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7lr\" (UniqueName: \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.296727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.296700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.298724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.298694 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.403788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.394092 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7lr\" (UniqueName: \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\") pod \"payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.412028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.411995 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d3b6ce-d0f1-4313-8e18-907fc389745a" path="/var/lib/kubelet/pods/46d3b6ce-d0f1-4313-8e18-907fc389745a/volumes" Oct 29 22:12:33.600529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.600443 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:33.909799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:33.909776 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:12:33.910441 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:12:33.910413 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb4b42e_a45d_4576_abb2_629dd96ff906.slice/crio-2e0387c53cea251e0250e5101a285dc7fea0021e8feb96ed36b2fb48713c0049 WatchSource:0}: Error finding container 2e0387c53cea251e0250e5101a285dc7fea0021e8feb96ed36b2fb48713c0049: Status 404 returned error can't find the container with id 2e0387c53cea251e0250e5101a285dc7fea0021e8feb96ed36b2fb48713c0049 Oct 29 22:12:34.567772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:34.567732 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" event={"ID":"dbb4b42e-a45d-4576-abb2-629dd96ff906","Type":"ContainerStarted","Data":"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea"} Oct 29 22:12:34.568208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:34.567795 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" event={"ID":"dbb4b42e-a45d-4576-abb2-629dd96ff906","Type":"ContainerStarted","Data":"2e0387c53cea251e0250e5101a285dc7fea0021e8feb96ed36b2fb48713c0049"} Oct 29 22:12:34.568208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:34.567823 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:34.615716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:34.615662 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" podStartSLOduration=2.6156477000000002 podStartE2EDuration="2.6156477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:34.611987896 +0000 UTC m=+1813.460464743" watchObservedRunningTime="2026-10-29 22:12:34.6156477 +0000 UTC m=+1813.464124547" Oct 29 22:12:41.423728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:41.423689 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 22:12:41.424327 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:41.423922 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" containerID="cri-o://d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db" gracePeriod=30 Oct 29 22:12:41.885073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:41.885036 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:12:41.886812 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:41.886745 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" podUID="ef07284b-1a12-49e3-a536-9c7692005c9b" containerName="maas-api" containerID="cri-o://84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4" gracePeriod=30 Oct 29 22:12:42.553922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.553873 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:12:42.707081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.704951 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\") pod \"ef07284b-1a12-49e3-a536-9c7692005c9b\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " Oct 29 22:12:42.707081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.705004 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\" (UniqueName: \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\") pod \"ef07284b-1a12-49e3-a536-9c7692005c9b\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " Oct 29 22:12:42.707081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.705117 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\") pod \"ef07284b-1a12-49e3-a536-9c7692005c9b\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " Oct 29 22:12:42.707081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.705253 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\") pod \"ef07284b-1a12-49e3-a536-9c7692005c9b\" (UID: \"ef07284b-1a12-49e3-a536-9c7692005c9b\") " Oct 29 22:12:42.711666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.711627 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs" pod "ef07284b-1a12-49e3-a536-9c7692005c9b" (UID: "ef07284b-1a12-49e3-a536-9c7692005c9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:42.712233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.712208 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca" pod "ef07284b-1a12-49e3-a536-9c7692005c9b" (UID: "ef07284b-1a12-49e3-a536-9c7692005c9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:42.715993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.715652 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db" pod "ef07284b-1a12-49e3-a536-9c7692005c9b" (UID: "ef07284b-1a12-49e3-a536-9c7692005c9b"). InnerVolumeSpecName "kube-api-access-tg4db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:42.739972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.738499 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls" pod "ef07284b-1a12-49e3-a536-9c7692005c9b" (UID: "ef07284b-1a12-49e3-a536-9c7692005c9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:42.807735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.807693 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07284b-1a12-49e3-a536-9c7692005c9b-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:42.807735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.807737 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4db\" (UniqueName: \"kubernetes.io/projected/ef07284b-1a12-49e3-a536-9c7692005c9b-kube-api-access-tg4db\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:42.808106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.807753 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:42.808106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.807766 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef07284b-1a12-49e3-a536-9c7692005c9b-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:42.828017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.825855 4376 generic.go:356] "Generic (PLEG): container finished" podID="ef07284b-1a12-49e3-a536-9c7692005c9b" containerID="84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4" exitCode=0 Oct 29 22:12:42.828017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.826060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" event={"ID":"ef07284b-1a12-49e3-a536-9c7692005c9b","Type":"ContainerDied","Data":"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4"} Oct 29 22:12:42.828017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.826116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" event={"ID":"ef07284b-1a12-49e3-a536-9c7692005c9b","Type":"ContainerDied","Data":"01dcfc1be3ff67fcfbeaa0cb7cbb7a65a2eb69e5fec3f0920b58fc6b2e8492a0"} Oct 29 22:12:42.828017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.826144 4376 scope.go:122] "RemoveContainer" containerID="84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4" Oct 29 22:12:42.828017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.826301 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt" Oct 29 22:12:42.840519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.840491 4376 scope.go:122] "RemoveContainer" containerID="84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4" Oct 29 22:12:42.840898 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:42.840875 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4\": container with ID starting with 84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4 not found: ID does not exist" containerID="84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4" Oct 29 22:12:42.841066 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.840953 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4"} err="failed to get container status \"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4\": rpc error: code = NotFound desc = could not find container \"84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4\": container with ID starting with 84222991a72cc7e538fc645ff6837695e4e15db37199f07fa8ff3f954677ebe4 not found: ID does not exist" Oct 29 22:12:42.890145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.890115 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:12:42.893943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:42.893888 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-665b09-5c89dffff9-cc8bt"] Oct 29 22:12:43.273758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:43.273720 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef07284b-1a12-49e3-a536-9c7692005c9b" path="/var/lib/kubelet/pods/ef07284b-1a12-49e3-a536-9c7692005c9b/volumes" Oct 29 22:12:45.578338 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:45.578308 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:12:46.442947 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:46.442869 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:12:46.699285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.699216 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 22:12:46.785115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785085 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785153 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785207 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785346 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785415 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.785572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785455 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785495 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785526 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785559 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785579 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.785842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785598 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\") pod \"01627787-0bc7-467e-90c3-2e847b1fa504\" (UID: \"01627787-0bc7-467e-90c3-2e847b1fa504\") " Oct 29 22:12:46.785842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.785807 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.786306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:46.786434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786415 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.786505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786435 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01627787-0bc7-467e-90c3-2e847b1fa504-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.786505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786444 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.786505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786452 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.786505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.786496 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.788470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.788425 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "kube-api-access-vzvfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:46.788877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.788849 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:46.789441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.789411 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:46.791230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.791208 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy" pod "01627787-0bc7-467e-90c3-2e847b1fa504" (UID: "01627787-0bc7-467e-90c3-2e847b1fa504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.862057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.862013 4376 generic.go:356] "Generic (PLEG): container finished" podID="01627787-0bc7-467e-90c3-2e847b1fa504" containerID="d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db" exitCode=0 Oct 29 22:12:46.862270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.862073 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" Oct 29 22:12:46.862270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.862096 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" event={"ID":"01627787-0bc7-467e-90c3-2e847b1fa504","Type":"ContainerDied","Data":"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db"} Oct 29 22:12:46.862270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.862169 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd" event={"ID":"01627787-0bc7-467e-90c3-2e847b1fa504","Type":"ContainerDied","Data":"b264f37df41ab57ac4220eb859cbb34a9ad9c35dde7eb73ba5805229348d5244"} Oct 29 22:12:46.862270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.862201 4376 scope.go:122] "RemoveContainer" containerID="d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db" Oct 29 22:12:46.880415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.880383 4376 scope.go:122] "RemoveContainer" containerID="d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db" Oct 29 22:12:46.883994 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:12:46.881428 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db\": container with ID starting with d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db not found: ID does not exist" containerID="d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db" Oct 29 22:12:46.883994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.881471 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db"} err="failed to get container status \"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db\": rpc error: code = NotFound desc = could not find container \"d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db\": container with ID starting with d7053f591556e8a6c82ac43f0793809a32083dba2eafeeecea9cfc7200aad1db not found: ID does not exist" Oct 29 22:12:46.888242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.888197 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvfd\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-kube-api-access-vzvfd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.888422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.888252 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01627787-0bc7-467e-90c3-2e847b1fa504-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.888422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.888275 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.888422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.888288 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01627787-0bc7-467e-90c3-2e847b1fa504-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.888422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.888299 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01627787-0bc7-467e-90c3-2e847b1fa504-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.927832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.927788 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 22:12:46.932660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:46.932636 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-cbncd"] Oct 29 22:12:47.288345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:47.288306 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" path="/var/lib/kubelet/pods/01627787-0bc7-467e-90c3-2e847b1fa504/volumes" Oct 29 22:12:48.911339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.911305 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:12:48.912026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912010 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ef07284b-1a12-49e3-a536-9c7692005c9b" containerName="maas-api" Oct 29 22:12:48.912077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912027 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef07284b-1a12-49e3-a536-9c7692005c9b" containerName="maas-api" Oct 29 22:12:48.912077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912042 4376 cpu_manager.go:414] "containerMap: removing container" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" Oct 29 22:12:48.912077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912050 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" Oct 29 22:12:48.912077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912058 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:48.912077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912062 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:48.912234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912208 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef07284b-1a12-49e3-a536-9c7692005c9b" containerName="maas-api" Oct 29 22:12:48.912234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912216 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="01627787-0bc7-467e-90c3-2e847b1fa504" containerName="istio-proxy" Oct 29 22:12:48.912234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.912224 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2097b8-6e8f-4e66-be1a-0aee6587d7d6" containerName="istio-proxy" Oct 29 22:12:48.915687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.915668 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.944526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.944494 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:12:48.950162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.950128 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-15154b-openshift-default-dockercfg-znxtz\"" Oct 29 22:12:48.989572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989541 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989586 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989687 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989709 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989808 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.989977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989962 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.990022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.989993 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7zb\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:48.990103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:48.990089 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.102491 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103158 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103354 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103413 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7zb\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103685 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103738 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103832 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.103877 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.105318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.106041 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.106624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.107938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.107236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.123788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.113050 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.124106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.124069 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.129795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.129424 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.138869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.137624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7zb\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.148045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.143327 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\") pod \"e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.235932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.235251 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:49.623030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.622999 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:12:49.625859 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:12:49.625830 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fd011b_f743_4c5e_b61a_a3c62d5ed463.slice/crio-66d6ff3c7110c516e24a1c810da2e2cd1b6d821c44058c82a4938b16f814ce72 WatchSource:0}: Error finding container 66d6ff3c7110c516e24a1c810da2e2cd1b6d821c44058c82a4938b16f814ce72: Status 404 returned error can't find the container with id 66d6ff3c7110c516e24a1c810da2e2cd1b6d821c44058c82a4938b16f814ce72 Oct 29 22:12:49.628997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.628962 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:49.629121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.629046 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:49.629121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.629094 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:49.706739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.706706 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:12:49.707135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.707079 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" containerID="cri-o://9f15baf346e5765a66a1065ca9223a52807d2d9e36ea3f76ac0bd33615fa5b27" gracePeriod=30 Oct 29 22:12:49.943976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.943142 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerStarted","Data":"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49"} Oct 29 22:12:49.943976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:49.943214 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerStarted","Data":"66d6ff3c7110c516e24a1c810da2e2cd1b6d821c44058c82a4938b16f814ce72"} Oct 29 22:12:50.009928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.009636 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podStartSLOduration=2.009614812 podStartE2EDuration="2.009614812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:50.007453457 +0000 UTC m=+1828.855930304" watchObservedRunningTime="2026-10-29 22:12:50.009614812 +0000 UTC m=+1828.858091658" Oct 29 22:12:50.236357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.236323 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:50.237743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.237710 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:12:50.237890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.237767 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:12:50.618461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.618385 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:12:50.621975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.621944 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.639532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.639507 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-bef83b\"" Oct 29 22:12:50.640160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.640136 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-bef83b-dockercfg-lmlwg\"" Oct 29 22:12:50.641820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.641796 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:12:50.755946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.737731 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflkz\" (UniqueName: \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.830953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.830893 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.933668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.933575 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflkz\" (UniqueName: \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.933839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.933669 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.934999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.934967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.952683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.952646 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:12:50.980930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.962205 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:50.990377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.986638 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflkz\" (UniqueName: \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\") pod \"payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:50.990377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:50.989104 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:12:51.036975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.036876 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.037175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.037050 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxmf\" (UniqueName: \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.139217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.139171 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.139399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.139289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxmf\" (UniqueName: \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.139823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.139803 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.166119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.166082 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxmf\" (UniqueName: \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\") pod \"payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.237045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.237016 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:51.242034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.242005 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:51.324867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.324822 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:51.550247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.550218 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:12:51.550500 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:12:51.550477 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fd685a_3c40_483e_b6fc_5087ec6ce75b.slice/crio-f9531ea29457565874e0f1aca58c73850dac738ba6d25bb723dd6b45cc547611 WatchSource:0}: Error finding container f9531ea29457565874e0f1aca58c73850dac738ba6d25bb723dd6b45cc547611: Status 404 returned error can't find the container with id f9531ea29457565874e0f1aca58c73850dac738ba6d25bb723dd6b45cc547611 Oct 29 22:12:51.612943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.612886 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:12:51.616514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.616496 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.634403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.634362 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:12:51.638090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.638054 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.644218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.644186 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05687d-openshift-default-dockercfg-l7t95\"" Oct 29 22:12:51.646115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.646087 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:12:51.652263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.652237 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:12:51.668674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.668643 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:12:51.753458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753425 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.753553 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753517 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22w22\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.753642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753615 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.753749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753729 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.753806 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753790 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.753913 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753887 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.753975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.753986 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754179 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754238 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.754319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754275 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.754319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754317 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754347 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.754456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754364 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwc8\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754434 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.754626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754483 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.754626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.754506 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.855614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855534 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.855614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855580 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.855614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855611 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855681 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855717 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855801 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.855873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855862 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855888 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwc8\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.855949 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856012 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856051 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856089 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856103 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856180 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856200 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856226 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22w22\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856260 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856340 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856381 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856202 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856460 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856561 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856723 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856792 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.856954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.856882 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.858147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.858123 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.858751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.858726 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.859077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.859061 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.859130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.859113 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.882724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.882688 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.882893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.882813 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22w22\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.883022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.882998 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwc8\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.883071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.882995 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:51.931586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.931539 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:51.964460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:51.964423 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:52.026994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.026444 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" event={"ID":"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad","Type":"ContainerStarted","Data":"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469"} Oct 29 22:12:52.026994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.026512 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" event={"ID":"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad","Type":"ContainerStarted","Data":"408a81cb2cfe9276bccc5dd6c1ed21064044b1cc78546b52751909e0f6417f33"} Oct 29 22:12:52.028602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.028546 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:12:52.032563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.031988 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" event={"ID":"76fd685a-3c40-483e-b6fc-5087ec6ce75b","Type":"ContainerStarted","Data":"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b"} Oct 29 22:12:52.032563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.032049 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:52.032563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.032077 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" event={"ID":"76fd685a-3c40-483e-b6fc-5087ec6ce75b","Type":"ContainerStarted","Data":"f9531ea29457565874e0f1aca58c73850dac738ba6d25bb723dd6b45cc547611"} Oct 29 22:12:52.032563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.032127 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:12:52.032889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.032838 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:12:52.079993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.079929 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" podStartSLOduration=2.079895957 podStartE2EDuration="2.079895957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:52.075245932 +0000 UTC m=+1830.923722778" watchObservedRunningTime="2026-10-29 22:12:52.079895957 +0000 UTC m=+1830.928372804" Oct 29 22:12:52.124060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.123938 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" podStartSLOduration=2.123919838 podStartE2EDuration="2.123919838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:52.121627625 +0000 UTC m=+1830.970104473" watchObservedRunningTime="2026-10-29 22:12:52.123919838 +0000 UTC m=+1830.972396687" Oct 29 22:12:52.260308 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:12:52.260270 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c73c94d_3dc2_4c12_8331_434de7e0009d.slice/crio-7087d53201e414e414b4e0cddaf43d6893e3a0c35196961fb49b630daf3b46bd WatchSource:0}: Error finding container 7087d53201e414e414b4e0cddaf43d6893e3a0c35196961fb49b630daf3b46bd: Status 404 returned error can't find the container with id 7087d53201e414e414b4e0cddaf43d6893e3a0c35196961fb49b630daf3b46bd Oct 29 22:12:52.260463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.260378 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:12:52.263479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.263440 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.263605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.263523 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.263605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.263566 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.356997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.355877 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:12:52.398805 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.398754 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.398975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.398862 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.398975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.398926 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:12:52.858419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.852219 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:12:52.858419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:52.852642 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" containerID="cri-o://de975296a9fb122fad97e62ef21489b3c25bf590b60667dfc1a9bccebc2d7701" gracePeriod=30 Oct 29 22:12:53.146023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.141341 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" event={"ID":"3c73c94d-3dc2-4c12-8331-434de7e0009d","Type":"ContainerStarted","Data":"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03"} Oct 29 22:12:53.146023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.141416 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" event={"ID":"3c73c94d-3dc2-4c12-8331-434de7e0009d","Type":"ContainerStarted","Data":"7087d53201e414e414b4e0cddaf43d6893e3a0c35196961fb49b630daf3b46bd"} Oct 29 22:12:53.155934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.151695 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" event={"ID":"2081f513-c697-494f-9031-b854d97de08c","Type":"ContainerStarted","Data":"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd"} Oct 29 22:12:53.155934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.151757 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" event={"ID":"2081f513-c697-494f-9031-b854d97de08c","Type":"ContainerStarted","Data":"56d2adc6910708a1957ec2822adacf1001298222d8a9f4f898bb82b9f2213691"} Oct 29 22:12:53.278814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.278733 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podStartSLOduration=2.278713049 podStartE2EDuration="2.278713049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:53.277796036 +0000 UTC m=+1832.126272886" watchObservedRunningTime="2026-10-29 22:12:53.278713049 +0000 UTC m=+1832.127189895" Oct 29 22:12:53.280419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.280346 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podStartSLOduration=2.280328898 podStartE2EDuration="2.280328898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:53.215238164 +0000 UTC m=+1832.063715022" watchObservedRunningTime="2026-10-29 22:12:53.280328898 +0000 UTC m=+1832.128805748" Oct 29 22:12:53.908928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.906843 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:12:53.908928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.906926 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:12:53.932501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.932465 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:12:53.955966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.937129 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:53.955966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.937191 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:53.972958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.966285 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:12:53.976951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.975139 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:53.976951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:53.975219 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:54.934474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:54.933207 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:54.934474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:54.933282 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:54.987927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:54.987294 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:54.987927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:54.987356 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:55.207690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.207541 4376 generic.go:356] "Generic (PLEG): container finished" podID="789c0500-3aab-4019-bc74-835b516fd5eb" containerID="9f15baf346e5765a66a1065ca9223a52807d2d9e36ea3f76ac0bd33615fa5b27" exitCode=0 Oct 29 22:12:55.207690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.207609 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerDied","Data":"9f15baf346e5765a66a1065ca9223a52807d2d9e36ea3f76ac0bd33615fa5b27"} Oct 29 22:12:55.207690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.207668 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" event={"ID":"789c0500-3aab-4019-bc74-835b516fd5eb","Type":"ContainerDied","Data":"0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154"} Oct 29 22:12:55.207690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.207687 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0582cd978778a784dbfe90617ee8e715a7d15ad34ed930042ff07d431e878154" Oct 29 22:12:55.208093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.207700 4376 scope.go:122] "RemoveContainer" containerID="cfc04ac62d80faa7c6fb85d234916ab259f5eed19a6412ce56214bbcf031f4a7" Oct 29 22:12:55.221072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.221053 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:12:55.419019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.408229 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.419374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.419350 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.439193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.419559 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.439699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.439660 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.441217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.441188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:55.441945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.441923 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.452055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.452031 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.452277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.452258 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.452422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.452404 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.452581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.452565 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\") pod \"789c0500-3aab-4019-bc74-835b516fd5eb\" (UID: \"789c0500-3aab-4019-bc74-835b516fd5eb\") " Oct 29 22:12:55.476335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.476299 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789c0500-3aab-4019-bc74-835b516fd5eb-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.481108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.476576 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.481108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.476949 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.481108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.477119 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.508124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.504074 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.512826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.512785 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:55.512965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.512939 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.513163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.513136 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "kube-api-access-pbxlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.515679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.515651 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token" pod "789c0500-3aab-4019-bc74-835b516fd5eb" (UID: "789c0500-3aab-4019-bc74-835b516fd5eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.586932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586840 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxlz\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-kube-api-access-pbxlz\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.586932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586880 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789c0500-3aab-4019-bc74-835b516fd5eb-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.586932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586895 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.586932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586932 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.587372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586946 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.587372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586960 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.587372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586972 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789c0500-3aab-4019-bc74-835b516fd5eb-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.587372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.586984 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789c0500-3aab-4019-bc74-835b516fd5eb-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.932293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.932251 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:55.932566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.932319 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:55.991927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.991069 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:55.991927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:55.991133 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:56.216528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.216497 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq" Oct 29 22:12:56.372930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.372788 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:12:56.390198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.390162 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5d6fq"] Oct 29 22:12:56.932372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.932258 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:56.932372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.932311 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:56.983928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.981937 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:56.983928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:56.981996 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:57.291869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:57.291821 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" path="/var/lib/kubelet/pods/789c0500-3aab-4019-bc74-835b516fd5eb/volumes" Oct 29 22:12:57.932266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:57.932238 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:57.932381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:57.932293 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:57.983935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:57.982213 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:57.983935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:57.982287 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:58.309101 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.309059 4376 generic.go:356] "Generic (PLEG): container finished" podID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerID="de975296a9fb122fad97e62ef21489b3c25bf590b60667dfc1a9bccebc2d7701" exitCode=0 Oct 29 22:12:58.309623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.309202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerDied","Data":"de975296a9fb122fad97e62ef21489b3c25bf590b60667dfc1a9bccebc2d7701"} Oct 29 22:12:58.309623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.309262 4376 scope.go:122] "RemoveContainer" containerID="1ca3f3ba67b7ee4a57566bb9ca99c7637f7b449cf7cb8f5a8a1628c114197a59" Oct 29 22:12:58.409178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.409149 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:58.519691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519658 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.519866 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519731 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.519866 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519794 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519874 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519955 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.519966 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:58.520053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.520049 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.520179 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:58.520257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.520182 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.520274 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.520368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.520323 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\") pod \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\" (UID: \"762bb4d0-b5d7-47ba-8671-b71f3d7684e6\") " Oct 29 22:12:58.521158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521029 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:58.521294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521215 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:58.521681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521404 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.521681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521428 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.521681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521459 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.521681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521473 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.521681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.521540 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:58.523168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.523137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "kube-api-access-w86nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:58.524034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.524010 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:58.527290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.527252 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:58.527626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.527603 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy" pod "762bb4d0-b5d7-47ba-8671-b71f3d7684e6" (UID: "762bb4d0-b5d7-47ba-8671-b71f3d7684e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:58.611466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.611390 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:12:58.612393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612373 4376 cpu_manager.go:414] "containerMap: removing container" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.612494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612485 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.612552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612546 4376 cpu_manager.go:414] "containerMap: removing container" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:58.612620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612612 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:58.612679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612672 4376 cpu_manager.go:414] "containerMap: removing container" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:58.612722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612716 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:58.612774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612768 4376 cpu_manager.go:414] "containerMap: removing container" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.612823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.612817 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.613063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.613053 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.613145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.613137 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:58.613204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.613198 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" containerName="istio-proxy" Oct 29 22:12:58.616019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.616003 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.623805 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.623776 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.623982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.623966 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.624090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.624074 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86nf\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-kube-api-access-w86nf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.624200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.624187 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.624286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.624275 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bb4d0-b5d7-47ba-8671-b71f3d7684e6-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:58.687384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.687351 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-1e74c3-openshift-default-dockercfg-s8qfv\"" Oct 29 22:12:58.706639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.698062 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.781441 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.781516 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.781683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.781950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5h8s\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.782018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.782047 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.782285 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.782400 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.787934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.782452 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.883774 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.883894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5h8s\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.883952 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.883976 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.884088 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.884161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.884203 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.884283 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.885016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.884318 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.908840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.907247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.908840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.908045 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.924238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.923088 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.924238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.923379 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:58.924238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:58.923469 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.036929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.035756 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.036929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.035818 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:59.056988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.054914 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.056988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.055013 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:59.392671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.392601 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.403000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.402308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.403000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.402647 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5h8s\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.403000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.402727 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.419069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.419029 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" event={"ID":"762bb4d0-b5d7-47ba-8671-b71f3d7684e6","Type":"ContainerDied","Data":"a5ff3407454d6390648392d3f5471c1e261228d2d864ef1c3d3a2664cdaa834e"} Oct 29 22:12:59.419323 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.419307 4376 scope.go:122] "RemoveContainer" containerID="de975296a9fb122fad97e62ef21489b3c25bf590b60667dfc1a9bccebc2d7701" Oct 29 22:12:59.419555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.419541 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v" Oct 29 22:12:59.507594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.507562 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:12:59.513488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.513446 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-665b09-openshift-default-56d79469b9-gl97v"] Oct 29 22:12:59.598756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.598723 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:12:59.897111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.895964 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:12:59.897356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.897332 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="789c0500-3aab-4019-bc74-835b516fd5eb" containerName="istio-proxy" Oct 29 22:12:59.907702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.907671 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:12:59.907885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.907789 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:12:59.925629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.925604 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-76c1b7-serving-cert\"" Oct 29 22:12:59.941404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.932970 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.941656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.941622 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:12:59.995073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.995032 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.995204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.995103 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:12:59.996773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.996673 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:12:59.996853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.996789 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5cx\" (UniqueName: \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:12:59.996853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.996828 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:12:59.997004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.996922 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:12:59.998440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:12:59.998410 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:13:00.001695 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:00.001664 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553a8d1c_4d61_4341_a642_699027f3f480.slice/crio-6070ea76f02246874f4879e4840fa2e127ccae72ff411182cde272ab9d1ad225 WatchSource:0}: Error finding container 6070ea76f02246874f4879e4840fa2e127ccae72ff411182cde272ab9d1ad225: Status 404 returned error can't find the container with id 6070ea76f02246874f4879e4840fa2e127ccae72ff411182cde272ab9d1ad225 Oct 29 22:13:00.005393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.005360 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:00.005492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.005450 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:00.005572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.005502 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:00.098857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.098819 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.099085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.099005 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5cx\" (UniqueName: \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.099085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.099053 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.099263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.099121 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.099878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.099846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.102584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.102534 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.104363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.104340 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.159937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.143674 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5cx\" (UniqueName: \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\") pod \"maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.227519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.227225 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:00.432080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.431072 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerStarted","Data":"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e"} Oct 29 22:13:00.432080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.431128 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerStarted","Data":"6070ea76f02246874f4879e4840fa2e127ccae72ff411182cde272ab9d1ad225"} Oct 29 22:13:00.485890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.485823 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podStartSLOduration=2.485802311 podStartE2EDuration="2.485802311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:00.481128797 +0000 UTC m=+1839.329605646" watchObservedRunningTime="2026-10-29 22:13:00.485802311 +0000 UTC m=+1839.334279162" Oct 29 22:13:00.601599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.601510 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:13:00.763230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.763196 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:13:00.763460 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:00.763431 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9909e5_b774_4720_858b_1c6271e8ac5c.slice/crio-31c721ef159ef42afa4517d2bc6f6bebec335f16f8e8f92ceb809970f2f48782 WatchSource:0}: Error finding container 31c721ef159ef42afa4517d2bc6f6bebec335f16f8e8f92ceb809970f2f48782: Status 404 returned error can't find the container with id 31c721ef159ef42afa4517d2bc6f6bebec335f16f8e8f92ceb809970f2f48782 Oct 29 22:13:00.936412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.936338 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:13:00.936847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.936822 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:13:00.937550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.937527 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:13:00.969399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.969368 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:13:00.969666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.969644 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:13:00.970345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:00.970326 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:13:01.275123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.275088 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762bb4d0-b5d7-47ba-8671-b71f3d7684e6" path="/var/lib/kubelet/pods/762bb4d0-b5d7-47ba-8671-b71f3d7684e6/volumes" Oct 29 22:13:01.462992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.459844 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" event={"ID":"3b9909e5-b774-4720-858b-1c6271e8ac5c","Type":"ContainerStarted","Data":"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a"} Oct 29 22:13:01.462992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.459893 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" event={"ID":"3b9909e5-b774-4720-858b-1c6271e8ac5c","Type":"ContainerStarted","Data":"31c721ef159ef42afa4517d2bc6f6bebec335f16f8e8f92ceb809970f2f48782"} Oct 29 22:13:01.462992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.460387 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:01.516604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.516549 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" podStartSLOduration=2.120786366 podStartE2EDuration="2.516534206s" totalImagesPullingTime="395.74784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:01.513752893 +0000 UTC m=+1840.362229736" watchObservedRunningTime="2026-10-29 22:13:01.516534206 +0000 UTC m=+1840.365011075" Oct 29 22:13:01.604311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:01.604276 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:13:02.465458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:02.465418 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:13:02.466190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:02.466169 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:13:04.232991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:04.220086 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:13:04.417979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:04.410038 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:13:08.478209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:08.478164 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:13:09.708629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:09.708591 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:13:09.709131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:09.708870 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerName="maas-api" containerID="cri-o://800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e" gracePeriod=30 Oct 29 22:13:09.744267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:09.743778 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.144:8443/health\": dial tcp 10.134.0.144:8443: connect: connection refused" Oct 29 22:13:10.551054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.548875 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:13:10.672297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.672267 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\") pod \"36b69b7b-2089-4d73-9e40-63f58f78cf78\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " Oct 29 22:13:10.672454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.672426 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") pod \"36b69b7b-2089-4d73-9e40-63f58f78cf78\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " Oct 29 22:13:10.672583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.672566 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\" (UniqueName: \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\") pod \"36b69b7b-2089-4d73-9e40-63f58f78cf78\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " Oct 29 22:13:10.672748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.672727 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\") pod \"36b69b7b-2089-4d73-9e40-63f58f78cf78\" (UID: \"36b69b7b-2089-4d73-9e40-63f58f78cf78\") " Oct 29 22:13:10.673685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.673649 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca" pod "36b69b7b-2089-4d73-9e40-63f58f78cf78" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:10.674690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.674662 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls" pod "36b69b7b-2089-4d73-9e40-63f58f78cf78" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:10.674776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.674690 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs" pod "36b69b7b-2089-4d73-9e40-63f58f78cf78" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:10.674776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.674754 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7" pod "36b69b7b-2089-4d73-9e40-63f58f78cf78" (UID: "36b69b7b-2089-4d73-9e40-63f58f78cf78"). InnerVolumeSpecName "kube-api-access-5fkc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:10.780334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.776444 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b69b7b-2089-4d73-9e40-63f58f78cf78-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:10.780334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.776486 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:10.780334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.776501 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36b69b7b-2089-4d73-9e40-63f58f78cf78-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:10.780334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.776512 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fkc7\" (UniqueName: \"kubernetes.io/projected/36b69b7b-2089-4d73-9e40-63f58f78cf78-kube-api-access-5fkc7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:10.808068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.808023 4376 generic.go:356] "Generic (PLEG): container finished" podID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerID="800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e" exitCode=0 Oct 29 22:13:10.808675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.808648 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" Oct 29 22:13:10.810535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.808887 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" event={"ID":"36b69b7b-2089-4d73-9e40-63f58f78cf78","Type":"ContainerDied","Data":"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e"} Oct 29 22:13:10.810535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.810073 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8" event={"ID":"36b69b7b-2089-4d73-9e40-63f58f78cf78","Type":"ContainerDied","Data":"e4bc7577623ab5728d9ed298877c760b76bcd104da1468c1951702989e178052"} Oct 29 22:13:10.810535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.810112 4376 scope.go:122] "RemoveContainer" containerID="800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e" Oct 29 22:13:10.857986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.857956 4376 scope.go:122] "RemoveContainer" containerID="800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e" Oct 29 22:13:10.858348 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:13:10.858324 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e\": container with ID starting with 800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e not found: ID does not exist" containerID="800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e" Oct 29 22:13:10.858421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.858358 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e"} err="failed to get container status \"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e\": rpc error: code = NotFound desc = could not find container \"800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e\": container with ID starting with 800f2633f49777cce5eeacea9981c06df930012a7700e8ef3466199bab6d909e not found: ID does not exist" Oct 29 22:13:10.908775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.908739 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:13:10.931837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:10.931809 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7721b3-846ccc44-677c8"] Oct 29 22:13:11.275195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:11.275162 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" path="/var/lib/kubelet/pods/36b69b7b-2089-4d73-9e40-63f58f78cf78/volumes" Oct 29 22:13:16.199126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.199086 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:13:16.199889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.199871 4376 cpu_manager.go:414] "containerMap: removing container" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerName="maas-api" Oct 29 22:13:16.199889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.199888 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerName="maas-api" Oct 29 22:13:16.200076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.200062 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b69b7b-2089-4d73-9e40-63f58f78cf78" containerName="maas-api" Oct 29 22:13:16.204074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.204056 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.226705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.226673 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:13:16.234100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.234071 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-bef83b-serving-cert\"" Oct 29 22:13:16.284397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.284359 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.284588 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.284428 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfdx\" (UniqueName: \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.284715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.284691 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.284772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.284757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.386186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.386143 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.386418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.386243 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.386418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.386379 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.386557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.386460 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfdx\" (UniqueName: \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.387227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.387182 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.389333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.389312 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.389462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.389424 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.413560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.413526 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfdx\" (UniqueName: \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\") pod \"maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.437462 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:13:16.437424 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:13:16.528933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.527501 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:16.922493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:16.922459 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:13:16.924308 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:16.924270 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540a0d3b_7030_49ec_bb4e_ba95228a7228.slice/crio-0f1173b65e580ff9e56e3245291ce4e6d306e59e44734e9f682916afed7c6eaa WatchSource:0}: Error finding container 0f1173b65e580ff9e56e3245291ce4e6d306e59e44734e9f682916afed7c6eaa: Status 404 returned error can't find the container with id 0f1173b65e580ff9e56e3245291ce4e6d306e59e44734e9f682916afed7c6eaa Oct 29 22:13:17.917212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:17.917101 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" event={"ID":"540a0d3b-7030-49ec-bb4e-ba95228a7228","Type":"ContainerStarted","Data":"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8"} Oct 29 22:13:17.917212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:17.917170 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" event={"ID":"540a0d3b-7030-49ec-bb4e-ba95228a7228","Type":"ContainerStarted","Data":"0f1173b65e580ff9e56e3245291ce4e6d306e59e44734e9f682916afed7c6eaa"} Oct 29 22:13:17.917728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:17.917526 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:17.965861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:17.965803 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" podStartSLOduration=1.600113865 podStartE2EDuration="1.965788015s" totalImagesPullingTime="365.67415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:17.9643799 +0000 UTC m=+1856.812856746" watchObservedRunningTime="2026-10-29 22:13:17.965788015 +0000 UTC m=+1856.814264860" Oct 29 22:13:18.149355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:18.149301 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:13:18.149943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:18.149893 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" containerID="cri-o://a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd" gracePeriod=30 Oct 29 22:13:19.761895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:19.753099 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:13:19.761895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:19.753350 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" podUID="70d79280-22ec-4971-a74a-96410efaeb96" containerName="maas-api" containerID="cri-o://074fb626f91153e5e16718b84b47915bcb665fa55b206cd8b9a44c589e83f5a2" gracePeriod=30 Oct 29 22:13:19.935091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:19.935059 4376 generic.go:356] "Generic (PLEG): container finished" podID="70d79280-22ec-4971-a74a-96410efaeb96" containerID="074fb626f91153e5e16718b84b47915bcb665fa55b206cd8b9a44c589e83f5a2" exitCode=0 Oct 29 22:13:19.935254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:19.935141 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" event={"ID":"70d79280-22ec-4971-a74a-96410efaeb96","Type":"ContainerDied","Data":"074fb626f91153e5e16718b84b47915bcb665fa55b206cd8b9a44c589e83f5a2"} Oct 29 22:13:20.342926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.337108 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:13:20.426926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.421531 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\") pod \"70d79280-22ec-4971-a74a-96410efaeb96\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " Oct 29 22:13:20.426926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.421584 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") pod \"70d79280-22ec-4971-a74a-96410efaeb96\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " Oct 29 22:13:20.426926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.421740 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\" (UniqueName: \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\") pod \"70d79280-22ec-4971-a74a-96410efaeb96\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " Oct 29 22:13:20.426926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.421964 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\") pod \"70d79280-22ec-4971-a74a-96410efaeb96\" (UID: \"70d79280-22ec-4971-a74a-96410efaeb96\") " Oct 29 22:13:20.426926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.423719 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca" pod "70d79280-22ec-4971-a74a-96410efaeb96" (UID: "70d79280-22ec-4971-a74a-96410efaeb96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:20.430924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.429759 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls" pod "70d79280-22ec-4971-a74a-96410efaeb96" (UID: "70d79280-22ec-4971-a74a-96410efaeb96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:20.430924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.430161 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs" pod "70d79280-22ec-4971-a74a-96410efaeb96" (UID: "70d79280-22ec-4971-a74a-96410efaeb96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:20.430924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.430567 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6" pod "70d79280-22ec-4971-a74a-96410efaeb96" (UID: "70d79280-22ec-4971-a74a-96410efaeb96"). InnerVolumeSpecName "kube-api-access-s5ss6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:20.524459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.524372 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d79280-22ec-4971-a74a-96410efaeb96-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:20.524459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.524420 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:20.524459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.524436 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ss6\" (UniqueName: \"kubernetes.io/projected/70d79280-22ec-4971-a74a-96410efaeb96-kube-api-access-s5ss6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:20.524459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.524449 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d79280-22ec-4971-a74a-96410efaeb96-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:20.599525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.599494 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:13:20.600223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.600209 4376 cpu_manager.go:414] "containerMap: removing container" podUID="70d79280-22ec-4971-a74a-96410efaeb96" containerName="maas-api" Oct 29 22:13:20.600281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.600224 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d79280-22ec-4971-a74a-96410efaeb96" containerName="maas-api" Oct 29 22:13:20.600362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.600353 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d79280-22ec-4971-a74a-96410efaeb96" containerName="maas-api" Oct 29 22:13:20.604051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.604035 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.621694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.621669 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-15154b-serving-cert\"" Oct 29 22:13:20.625421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.625395 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:13:20.727996 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.727954 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrqg\" (UniqueName: \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.728260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.728236 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.728353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.728271 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.728418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.728357 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.829843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.829752 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrqg\" (UniqueName: \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.830326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.829992 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.830326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.830015 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.830326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.830056 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.830791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.830766 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.832729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.832706 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.832729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.832720 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.892937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.892497 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrqg\" (UniqueName: \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\") pod \"maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.926687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.926392 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:20.995968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.990809 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" event={"ID":"70d79280-22ec-4971-a74a-96410efaeb96","Type":"ContainerDied","Data":"5ee9e9427e9b0cdc3a1d6484dc1c94d04f6f26d445a474a570f7e5bb305fbc88"} Oct 29 22:13:20.995968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.990966 4376 scope.go:122] "RemoveContainer" containerID="074fb626f91153e5e16718b84b47915bcb665fa55b206cd8b9a44c589e83f5a2" Oct 29 22:13:20.995968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:20.991205 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g" Oct 29 22:13:21.310434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:21.241063 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:13:21.336059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:21.336026 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b0939-5785f7c77f-2bm9g"] Oct 29 22:13:21.379647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:21.373818 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d79280-22ec-4971-a74a-96410efaeb96" path="/var/lib/kubelet/pods/70d79280-22ec-4971-a74a-96410efaeb96/volumes" Oct 29 22:13:21.605775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:21.605748 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:13:21.606292 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:21.606265 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd25ab2_2855_46a2_aed0_8b4abfad235e.slice/crio-a3589090be9aeadc68eb901d9fb0e413685cce677e88f03c199d88e16d89e1a1 WatchSource:0}: Error finding container a3589090be9aeadc68eb901d9fb0e413685cce677e88f03c199d88e16d89e1a1: Status 404 returned error can't find the container with id a3589090be9aeadc68eb901d9fb0e413685cce677e88f03c199d88e16d89e1a1 Oct 29 22:13:22.085926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:22.075923 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" event={"ID":"5fd25ab2-2855-46a2-aed0-8b4abfad235e","Type":"ContainerStarted","Data":"a3589090be9aeadc68eb901d9fb0e413685cce677e88f03c199d88e16d89e1a1"} Oct 29 22:13:23.365851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:23.365811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" event={"ID":"5fd25ab2-2855-46a2-aed0-8b4abfad235e","Type":"ContainerStarted","Data":"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca"} Oct 29 22:13:23.366502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:23.366478 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:23.442975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:23.430975 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" podStartSLOduration=3.100675125 podStartE2EDuration="3.430952606s" totalImagesPullingTime="330.277481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:23.421871796 +0000 UTC m=+1862.270348643" watchObservedRunningTime="2026-10-29 22:13:23.430952606 +0000 UTC m=+1862.279429452" Oct 29 22:13:24.129164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.127777 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:13:24.140621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.140585 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:13:24.195934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.195885 4376 scope.go:122] "RemoveContainer" containerID="9f15baf346e5765a66a1065ca9223a52807d2d9e36ea3f76ac0bd33615fa5b27" Oct 29 22:13:24.290837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.287794 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.290837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.288111 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.290837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.288347 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.290837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.288632 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.292208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.288715 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.292362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.292343 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.292752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.292680 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.292949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.292892 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.296920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.293208 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\") pod \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\" (UID: \"5b98a77c-450d-4a4e-b812-ad4f1dd81af5\") " Oct 29 22:13:24.331021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.330980 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:24.331313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.331288 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:24.331610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.331586 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:24.331780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.331748 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.334030 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.334363 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.336826 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.337282 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.338024 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.338430 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.339185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.338807 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.339930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.339803 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.444210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.424353 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:24.444210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.427812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:13:24.449092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.444682 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.449092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.444712 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:24.449092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.448921 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk" pod "5b98a77c-450d-4a4e-b812-ad4f1dd81af5" (UID: "5b98a77c-450d-4a4e-b812-ad4f1dd81af5"). InnerVolumeSpecName "kube-api-access-ndcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:24.459703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.458464 4376 generic.go:356] "Generic (PLEG): container finished" podID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerID="a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd" exitCode=0 Oct 29 22:13:24.459703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.458711 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerDied","Data":"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd"} Oct 29 22:13:24.459703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.458770 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" event={"ID":"5b98a77c-450d-4a4e-b812-ad4f1dd81af5","Type":"ContainerDied","Data":"b3251c1d64a227e276cbc450be05285187d0239c5a0095747b70b5a5efee3bf8"} Oct 29 22:13:24.459703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.458802 4376 scope.go:122] "RemoveContainer" containerID="a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd" Oct 29 22:13:24.459703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.458847 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7" Oct 29 22:13:24.478554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.478529 4376 scope.go:122] "RemoveContainer" containerID="138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3" Oct 29 22:13:24.499194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.499167 4376 scope.go:122] "RemoveContainer" containerID="a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd" Oct 29 22:13:24.499550 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:13:24.499518 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd\": container with ID starting with a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd not found: ID does not exist" containerID="a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd" Oct 29 22:13:24.499653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.499558 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd"} err="failed to get container status \"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd\": rpc error: code = NotFound desc = could not find container \"a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd\": container with ID starting with a6120648d95128689462b550ae75a7999932f10ab84cb89fd726633d50f74afd not found: ID does not exist" Oct 29 22:13:24.499653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.499585 4376 scope.go:122] "RemoveContainer" containerID="138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3" Oct 29 22:13:24.499928 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:13:24.499884 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3\": container with ID starting with 138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3 not found: ID does not exist" containerID="138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3" Oct 29 22:13:24.500018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.499936 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3"} err="failed to get container status \"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3\": rpc error: code = NotFound desc = could not find container \"138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3\": container with ID starting with 138267c07ed411a1481eee3a97cc0cad552f33ca42c1e610d9fc7f23d35bd3b3 not found: ID does not exist" Oct 29 22:13:24.521670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.521638 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:13:24.529707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.529673 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7721b3-openshift-default-fb967cc77-8gpz7"] Oct 29 22:13:24.546662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:24.546593 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcdk\" (UniqueName: \"kubernetes.io/projected/5b98a77c-450d-4a4e-b812-ad4f1dd81af5-kube-api-access-ndcdk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:25.276742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:25.276708 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" path="/var/lib/kubelet/pods/5b98a77c-450d-4a4e-b812-ad4f1dd81af5/volumes" Oct 29 22:13:29.392778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:29.392741 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:13:29.471361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:29.471325 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:13:30.526935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.526825 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:13:30.538958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.528145 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:30.538958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.528173 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:30.538958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.528446 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:30.558485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.558445 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:13:30.558793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.558776 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.578116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.578087 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-1e74c3-serving-cert\"" Oct 29 22:13:30.780263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.780080 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.786281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.786242 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.792927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.786760 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.799962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.795400 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgr8\" (UniqueName: \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.897565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.897533 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.897753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.897607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgr8\" (UniqueName: \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.897753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.897712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.897853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.897816 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.898416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.898387 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.900237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.900215 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.900330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.900303 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.924769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.924740 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgr8\" (UniqueName: \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\") pod \"maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:30.949517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:30.949484 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:31.262308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:31.262269 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:13:31.262648 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:31.262620 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fff7a2_f346_408c_83ee_b8a591643a9f.slice/crio-00e0f5a0c9cf9ef5f209642059695465ae6fe82dd95286f3908b9821cebd5691 WatchSource:0}: Error finding container 00e0f5a0c9cf9ef5f209642059695465ae6fe82dd95286f3908b9821cebd5691: Status 404 returned error can't find the container with id 00e0f5a0c9cf9ef5f209642059695465ae6fe82dd95286f3908b9821cebd5691 Oct 29 22:13:31.559988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:31.559891 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" event={"ID":"a9fff7a2-f346-408c-83ee-b8a591643a9f","Type":"ContainerStarted","Data":"00e0f5a0c9cf9ef5f209642059695465ae6fe82dd95286f3908b9821cebd5691"} Oct 29 22:13:32.586930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:32.575111 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" event={"ID":"a9fff7a2-f346-408c-83ee-b8a591643a9f","Type":"ContainerStarted","Data":"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c"} Oct 29 22:13:32.586930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:32.576647 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:32.639963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:32.639240 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" podStartSLOduration=2.230746327 podStartE2EDuration="2.639219052s" totalImagesPullingTime="408.472725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:32.637976047 +0000 UTC m=+1871.486452895" watchObservedRunningTime="2026-10-29 22:13:32.639219052 +0000 UTC m=+1871.487695899" Oct 29 22:13:34.393199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.393156 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" podUID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" containerName="payload-processing" containerID="cri-o://e2c1678c8d2613c3eedb17399fd160e72ad2407cd10eafc87ec9d51e633c3d3f" gracePeriod=25 Oct 29 22:13:34.597459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.596347 4376 generic.go:356] "Generic (PLEG): container finished" podID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" containerID="e2c1678c8d2613c3eedb17399fd160e72ad2407cd10eafc87ec9d51e633c3d3f" exitCode=0 Oct 29 22:13:34.597459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.596682 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" event={"ID":"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca","Type":"ContainerDied","Data":"e2c1678c8d2613c3eedb17399fd160e72ad2407cd10eafc87ec9d51e633c3d3f"} Oct 29 22:13:34.686194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.686168 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:13:34.818140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.818104 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\") pod \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " Oct 29 22:13:34.818320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.818215 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\" (UniqueName: \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\") pod \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\" (UID: \"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca\") " Oct 29 22:13:34.818551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.818522 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume" pod "9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" (UID: "9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:34.819142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.819123 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.820357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.820333 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g" pod "9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" (UID: "9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca"). InnerVolumeSpecName "kube-api-access-wgw4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:34.920532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:34.920446 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgw4g\" (UniqueName: \"kubernetes.io/projected/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca-kube-api-access-wgw4g\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:35.605292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.605213 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" Oct 29 22:13:35.605695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.605212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb" event={"ID":"9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca","Type":"ContainerDied","Data":"ec3541cb47cfc2afe1db3e6f23b38e8a267979c4c1a8ccfd5201416ff3807d08"} Oct 29 22:13:35.605695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.605361 4376 scope.go:122] "RemoveContainer" containerID="e2c1678c8d2613c3eedb17399fd160e72ad2407cd10eafc87ec9d51e633c3d3f" Oct 29 22:13:35.668560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.668519 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:13:35.679804 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.679775 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4c5549-96644f686-zpwwb"] Oct 29 22:13:35.907227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.907142 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:13:35.908371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908341 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:35.908371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908370 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:35.908515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908408 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" containerName="payload-processing" Oct 29 22:13:35.908515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908416 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" containerName="payload-processing" Oct 29 22:13:35.908592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908578 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" containerName="payload-processing" Oct 29 22:13:35.908631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.908602 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b98a77c-450d-4a4e-b812-ad4f1dd81af5" containerName="istio-proxy" Oct 29 22:13:35.914160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.914139 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.936620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.936592 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:13:35.944198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.944142 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-05687d\"/\"kube-root-ca.crt\"" Oct 29 22:13:35.944198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.944156 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-05687d\"/\"default-dockercfg-z7w8f\"" Oct 29 22:13:35.944198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.944186 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-05687d\"/\"openshift-service-ca.crt\"" Oct 29 22:13:35.944416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.944238 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-05687d\"/\"auth-test-model-05687d-kserve-self-signed-certs\"" Oct 29 22:13:35.945075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.944886 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.945075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.945041 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.945223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.945167 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.945263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.945240 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb6p5\" (UniqueName: \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.945324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.945308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:35.945542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:35.945523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.046763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.046725 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.046769 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6p5\" (UniqueName: \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.046823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047027 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047157 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047284 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047290 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047383 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.047653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.047631 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.049613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.049586 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.049862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.049844 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.073897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.073867 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6p5\" (UniqueName: \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\") pod \"auth-test-model-05687d-kserve-5c4d85cc65-fc6zx\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.244928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.244465 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:36.679692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.679660 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:13:36.680233 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:36.680208 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod580bb07a_d7d7_4d59_81d6_620c68eeb26f.slice/crio-2764af5c1166a3e756d438a9501548282c33b9561e7882eee8726c00a37e335e WatchSource:0}: Error finding container 2764af5c1166a3e756d438a9501548282c33b9561e7882eee8726c00a37e335e: Status 404 returned error can't find the container with id 2764af5c1166a3e756d438a9501548282c33b9561e7882eee8726c00a37e335e Oct 29 22:13:36.779189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.776415 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:13:36.826959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.818152 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:13:36.826959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.818321 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873340 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqkl\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873490 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873532 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873608 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873639 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873689 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873729 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873862 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.873981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.873946 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975283 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975489 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqkl\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975668 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975698 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975747 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.975942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.976013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.975996 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.976112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.976092 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.976186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.976163 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.976509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.976491 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.977584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.977560 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:36.978127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:36.978107 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:37.001341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.001316 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:37.001341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.001335 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqkl\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:37.147191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.147155 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:37.279396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.279220 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca" path="/var/lib/kubelet/pods/9a6e1ed7-d38b-4780-b5aa-ac62de47d8ca/volumes" Oct 29 22:13:37.475192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.475159 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:13:37.477497 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:13:37.477461 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1093f24b_6824_47b6_a2fb_cedb867475b3.slice/crio-9bc42efd15147a857fb8c22729be934cdf1e00d2e16f2f345b8a1f1718bbca14 WatchSource:0}: Error finding container 9bc42efd15147a857fb8c22729be934cdf1e00d2e16f2f345b8a1f1718bbca14: Status 404 returned error can't find the container with id 9bc42efd15147a857fb8c22729be934cdf1e00d2e16f2f345b8a1f1718bbca14 Oct 29 22:13:37.480930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.480872 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:37.481048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.480978 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:37.481048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.481021 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:13:37.628046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.628007 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" event={"ID":"1093f24b-6824-47b6-a2fb-cedb867475b3","Type":"ContainerStarted","Data":"9bc42efd15147a857fb8c22729be934cdf1e00d2e16f2f345b8a1f1718bbca14"} Oct 29 22:13:37.629792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.629764 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" event={"ID":"580bb07a-d7d7-4d59-81d6-620c68eeb26f","Type":"ContainerStarted","Data":"82347169bb4e9248056be9ada301b8be1e99f32e6f50f4c01d4b37d0646b24ee"} Oct 29 22:13:37.629965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.629804 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" event={"ID":"580bb07a-d7d7-4d59-81d6-620c68eeb26f","Type":"ContainerStarted","Data":"2764af5c1166a3e756d438a9501548282c33b9561e7882eee8726c00a37e335e"} Oct 29 22:13:37.676777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:37.676704 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" podStartSLOduration=2.676683532 podStartE2EDuration="2.676683532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:37.674351731 +0000 UTC m=+1876.522828578" watchObservedRunningTime="2026-10-29 22:13:37.676683532 +0000 UTC m=+1876.525160386" Oct 29 22:13:38.263041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:38.262999 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:38.648182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:38.648021 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" event={"ID":"1093f24b-6824-47b6-a2fb-cedb867475b3","Type":"ContainerStarted","Data":"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952"} Oct 29 22:13:38.759792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:38.759710 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podStartSLOduration=2.759687049 podStartE2EDuration="2.759687049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:38.725159463 +0000 UTC m=+1877.573636308" watchObservedRunningTime="2026-10-29 22:13:38.759687049 +0000 UTC m=+1877.608163897" Oct 29 22:13:39.157275 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:39.157236 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:39.157696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:39.157669 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:13:39.157892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:39.157860 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 29 22:13:39.604054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:39.604020 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:13:40.153147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:40.150642 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:13:40.153147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:40.150801 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 29 22:13:41.153475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:41.153430 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:13:41.154179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:41.154137 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 29 22:13:42.155396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:42.155349 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:13:42.157325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:42.157286 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 29 22:13:43.154701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:43.154668 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:13:43.155042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:43.155000 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 29 22:13:44.152972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:44.152934 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:44.153437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:44.153423 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:44.154153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:44.154133 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:13:46.245921 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:46.245878 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:46.260952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:46.260925 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:13:46.437173 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:13:46.437133 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:13:46.738848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:13:46.738818 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:14:05.538834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.538775 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:14:05.646307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.646270 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.649311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.649286 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:14:05.675391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.675349 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4afaba-openshift-default-dockercfg-zzg8x\"" Oct 29 22:14:05.791072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.790970 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.791378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.791353 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.791508 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.791490 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.805294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.805247 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.873067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.873027 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt42b\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.913441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.913393 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.913750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.913729 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.914083 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.914061 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:05.959014 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:05.914265 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.206702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.206646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.210327 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210273 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt42b\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.210829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210341 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.210829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210404 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.210829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210624 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.210829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210674 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.211107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210847 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.211107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.210985 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.211107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.211030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.218003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.211851 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.218228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.212858 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.218372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.217388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.218492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.218128 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.218581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.218381 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.224824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.224795 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.227810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.227774 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.267100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.267057 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt42b\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.267960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.267929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\") pod \"e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.309193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.309156 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:06.693157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.693128 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:14:06.693503 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:14:06.693433 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58c4821_b7f6_4b5b_8215_48c20db90b6b.slice/crio-4e392ed58d33ae04039e430c6dbe1011a25d192939a29886eea6faab8076bc9f WatchSource:0}: Error finding container 4e392ed58d33ae04039e430c6dbe1011a25d192939a29886eea6faab8076bc9f: Status 404 returned error can't find the container with id 4e392ed58d33ae04039e430c6dbe1011a25d192939a29886eea6faab8076bc9f Oct 29 22:14:06.696142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.696104 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:06.696228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.696195 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:06.696265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:06.696238 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:07.275273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.275231 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerStarted","Data":"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c"} Oct 29 22:14:07.275494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.275316 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerStarted","Data":"4e392ed58d33ae04039e430c6dbe1011a25d192939a29886eea6faab8076bc9f"} Oct 29 22:14:07.309916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.309877 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:07.316544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.316517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:07.332390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.332326 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podStartSLOduration=2.332306463 podStartE2EDuration="2.332306463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:07.331737199 +0000 UTC m=+1906.180214045" watchObservedRunningTime="2026-10-29 22:14:07.332306463 +0000 UTC m=+1906.180783311" Oct 29 22:14:07.766119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.766085 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 22:14:07.766544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:07.766286 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" containerID="cri-o://6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727" gracePeriod=30 Oct 29 22:14:08.275803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:08.275772 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:08.276914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:08.276877 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:14:13.075958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.075932 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 22:14:13.092005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.091968 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092175 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:13.092224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092205 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092246 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092365 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092421 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092458 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092498 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:13.092781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092544 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092582 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\") pod \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\" (UID: \"a1f10461-40a5-4563-8bcc-e2ab4780f76a\") " Oct 29 22:14:13.092781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.092591 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:13.094071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094040 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:13.094245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094214 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.094325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094251 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.094325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094265 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.094325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094305 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.094940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.094506 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:14:13.096616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.096434 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:13.096616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.096472 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "kube-api-access-4fcg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:13.097185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.097157 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:14:13.097302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.097282 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token" pod "a1f10461-40a5-4563-8bcc-e2ab4780f76a" (UID: "a1f10461-40a5-4563-8bcc-e2ab4780f76a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:13.195990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.195955 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcg8\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-kube-api-access-4fcg8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.195990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.195985 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.195990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.195995 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.196222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.196003 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.196222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.196011 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f10461-40a5-4563-8bcc-e2ab4780f76a-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:13.475532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.475494 4376 generic.go:356] "Generic (PLEG): container finished" podID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerID="6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727" exitCode=0 Oct 29 22:14:13.475738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.475537 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" event={"ID":"a1f10461-40a5-4563-8bcc-e2ab4780f76a","Type":"ContainerDied","Data":"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727"} Oct 29 22:14:13.475738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.475576 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" Oct 29 22:14:13.475738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.475608 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p" event={"ID":"a1f10461-40a5-4563-8bcc-e2ab4780f76a","Type":"ContainerDied","Data":"ce6016effce2e996f71caa43fe096224b3cebd108ff1a0a4025f65ce3ac833d9"} Oct 29 22:14:13.475738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.475639 4376 scope.go:122] "RemoveContainer" containerID="6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727" Oct 29 22:14:13.490085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.490057 4376 scope.go:122] "RemoveContainer" containerID="6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727" Oct 29 22:14:13.490477 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:13.490449 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727\": container with ID starting with 6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727 not found: ID does not exist" containerID="6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727" Oct 29 22:14:13.490574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.490494 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727"} err="failed to get container status \"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727\": rpc error: code = NotFound desc = could not find container \"6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727\": container with ID starting with 6499bd5958929e2e5c6dc50688a09ddfe9b498ecbe4655309f6890f1c34b9727 not found: ID does not exist" Oct 29 22:14:13.544895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.544850 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 22:14:13.558246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:13.558212 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-xhm5p"] Oct 29 22:14:15.275403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:15.275369 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" path="/var/lib/kubelet/pods/a1f10461-40a5-4563-8bcc-e2ab4780f76a/volumes" Oct 29 22:14:16.437156 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:16.437114 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:14:16.507817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.507784 4376 generic.go:356] "Generic (PLEG): container finished" podID="553a8d1c-4d61-4341-a642-699027f3f480" containerID="3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e" exitCode=0 Oct 29 22:14:16.508079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.507863 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerDied","Data":"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e"} Oct 29 22:14:16.508296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.508280 4376 scope.go:122] "RemoveContainer" containerID="3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e" Oct 29 22:14:16.509832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.509802 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:16.509884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.509872 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:16.509958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.509947 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:16.807293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.807259 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-c7cdde"] Oct 29 22:14:16.808010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.807995 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" Oct 29 22:14:16.808064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.808010 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" Oct 29 22:14:16.808143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.808133 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f10461-40a5-4563-8bcc-e2ab4780f76a" containerName="istio-proxy" Oct 29 22:14:16.810991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.810971 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:16.828958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.828932 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:14:16.832446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.832409 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-c7cdde"] Oct 29 22:14:16.931073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:16.931040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmhhj\" (UniqueName: \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\") pod \"test-curl-7761-c7cdde\" (UID: \"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb\") " pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:17.032596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.032558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmhhj\" (UniqueName: \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\") pod \"test-curl-7761-c7cdde\" (UID: \"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb\") " pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:17.058587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.058509 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmhhj\" (UniqueName: \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\") pod \"test-curl-7761-c7cdde\" (UID: \"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb\") " pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:17.124722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.124683 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:17.432155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.432120 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-c7cdde"] Oct 29 22:14:17.465995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.465957 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:14:17.517254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.517224 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerStarted","Data":"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300"} Oct 29 22:14:17.519010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.518988 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-c7cdde" event={"ID":"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb","Type":"ContainerStarted","Data":"002f43615ffb84d1b13f862706e8c3ac417cb7f9d9555cc765228b014f8a4908"} Oct 29 22:14:17.599846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.599814 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:14:17.601470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.601441 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:17.601579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:17.601504 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:18.527710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:18.527668 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-c7cdde" event={"ID":"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb","Type":"ContainerStarted","Data":"deca664bbce4bf4136df5f014f5d6ce66fcf6b4046291d284d2810dd7307bf6b"} Oct 29 22:14:18.572869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:18.572818 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7761-c7cdde" podStartSLOduration=2.572803506 podStartE2EDuration="2.572803506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:18.572068619 +0000 UTC m=+1917.420545468" watchObservedRunningTime="2026-10-29 22:14:18.572803506 +0000 UTC m=+1917.421280351" Oct 29 22:14:18.599796 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:18.599759 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:18.599995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:18.599817 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:19.065489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.060965 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:14:19.066977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.066610 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.099959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.095824 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8d569e-dockercfg-hw2nn\"" Oct 29 22:14:19.137928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.123024 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8d569e\"" Oct 29 22:14:19.242988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.219477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.242988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.219573 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd29m\" (UniqueName: \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.363125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.363033 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.371457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.371415 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd29m\" (UniqueName: \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.387126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.364897 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.406070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.405071 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd29m\" (UniqueName: \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\") pod \"payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:19.421394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.421360 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:14:19.599607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.599575 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:19.600061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.599630 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:19.657980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:19.653227 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:20.047930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.047765 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:14:20.071692 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:14:20.071659 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723965e_5757_49b5_a066_865ce2b12406.slice/crio-9dcdf8f8cf06e3fd26e31d409512bbc105bcc2d9906aea54bc6784d59eb64473 WatchSource:0}: Error finding container 9dcdf8f8cf06e3fd26e31d409512bbc105bcc2d9906aea54bc6784d59eb64473: Status 404 returned error can't find the container with id 9dcdf8f8cf06e3fd26e31d409512bbc105bcc2d9906aea54bc6784d59eb64473 Oct 29 22:14:20.193853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.193820 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-c7cdde"] Oct 29 22:14:20.202586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.202548 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-c7cdde"] Oct 29 22:14:20.568875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.568834 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" event={"ID":"8723965e-5757-49b5-a066-865ce2b12406","Type":"ContainerStarted","Data":"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c"} Oct 29 22:14:20.569075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.568968 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7761-c7cdde" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" containerName="test-curl-7761-c7cdde" containerID="cri-o://deca664bbce4bf4136df5f014f5d6ce66fcf6b4046291d284d2810dd7307bf6b" gracePeriod=2 Oct 29 22:14:20.569143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.569082 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:20.569143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.569110 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" event={"ID":"8723965e-5757-49b5-a066-865ce2b12406","Type":"ContainerStarted","Data":"9dcdf8f8cf06e3fd26e31d409512bbc105bcc2d9906aea54bc6784d59eb64473"} Oct 29 22:14:20.584011 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:20.583971 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-c7cdde\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:20.599973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.599942 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:20.600366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.599995 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:20.631445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:20.631386 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" podStartSLOduration=2.631373102 podStartE2EDuration="2.631373102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:20.629830066 +0000 UTC m=+1919.478306917" watchObservedRunningTime="2026-10-29 22:14:20.631373102 +0000 UTC m=+1919.479849947" Oct 29 22:14:20.657928 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:20.657200 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-c7cdde\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:21.214110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.214075 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:14:21.214834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.214815 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" containerName="test-curl-7761-c7cdde" Oct 29 22:14:21.214834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.214832 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" containerName="test-curl-7761-c7cdde" Oct 29 22:14:21.215015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.214977 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" containerName="test-curl-7761-c7cdde" Oct 29 22:14:21.217621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.217600 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.237307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.237278 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0a3d53-openshift-default-dockercfg-46flq\"" Oct 29 22:14:21.237720 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:21.233056 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-c7cdde\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:21.279388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.279347 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.279674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.279652 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.279844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.279826 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.280395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.280009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl7q\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.280719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.280684 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.280952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.280929 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.281101 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.281083 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.281273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.281256 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.281451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.281430 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.298253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.298222 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:14:21.311353 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:21.311310 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-c7cdde\" is forbidden: User \"system:node:ip-10-0-141-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-100.ec2.internal' and this object" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:21.418927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.393812 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.427084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.427038 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.457930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.431015 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.458724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.458690 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.459054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.459030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.459206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.459191 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.459384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.459369 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.483016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.463024 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl7q\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.483490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.483453 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.501926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.493308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.573937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.564354 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.573937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.459540 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.573937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.458356 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.573937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.565128 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.736980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.645790 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:21.756050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.738154 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:21.756050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.720374 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.780768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.780717 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.781043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.780825 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.783337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.783123 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl7q\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\") pod \"e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:21.918422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.918379 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0a3d53-openshift-default-dockercfg-46flq\"" Oct 29 22:14:21.920094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:21.918684 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:22.382949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.382918 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:14:22.383094 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:14:22.383049 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e732d9_1fa1_43bb_819d_0da8a0b35787.slice/crio-0c1ed29f2cbe6ac708404c2db19f5523a5760e11b93d7be3331ec83c3d186788 WatchSource:0}: Error finding container 0c1ed29f2cbe6ac708404c2db19f5523a5760e11b93d7be3331ec83c3d186788: Status 404 returned error can't find the container with id 0c1ed29f2cbe6ac708404c2db19f5523a5760e11b93d7be3331ec83c3d186788 Oct 29 22:14:22.386585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.386535 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:22.386698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.386650 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:22.386774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.386693 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:22.599650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.599621 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:22.599788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.599673 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:22.731017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.729817 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerStarted","Data":"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e"} Oct 29 22:14:22.731017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.729873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerStarted","Data":"0c1ed29f2cbe6ac708404c2db19f5523a5760e11b93d7be3331ec83c3d186788"} Oct 29 22:14:22.732876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.732394 4376 generic.go:356] "Generic (PLEG): container finished" podID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" containerID="deca664bbce4bf4136df5f014f5d6ce66fcf6b4046291d284d2810dd7307bf6b" exitCode=137 Oct 29 22:14:22.802004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.801653 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podStartSLOduration=1.801633217 podStartE2EDuration="1.801633217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:22.800935722 +0000 UTC m=+1921.649412568" watchObservedRunningTime="2026-10-29 22:14:22.801633217 +0000 UTC m=+1921.650110064" Oct 29 22:14:22.921456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.921405 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:22.968932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:22.966190 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:23.103261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.103179 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\" (UniqueName: \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\") pod \"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb\" (UID: \"d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb\") " Oct 29 22:14:23.105325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.105299 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj" pod "d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" (UID: "d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb"). InnerVolumeSpecName "kube-api-access-dmhhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:23.205519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.205476 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmhhj\" (UniqueName: \"kubernetes.io/projected/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb-kube-api-access-dmhhj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:23.274530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.274492 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb" path="/var/lib/kubelet/pods/d83754a9-3bfc-4ffa-8ab2-c2dd5268a6eb/volumes" Oct 29 22:14:23.606926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.603944 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:14:23.606926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.604011 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:14:23.743758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.743725 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-c7cdde" Oct 29 22:14:23.744008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.743763 4376 scope.go:122] "RemoveContainer" containerID="deca664bbce4bf4136df5f014f5d6ce66fcf6b4046291d284d2810dd7307bf6b" Oct 29 22:14:23.925872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:23.925778 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:24.605103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:24.605071 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:14:24.605516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:24.605500 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:14:24.606286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:24.606263 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:14:24.758182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:24.757027 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:24.758182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:24.758013 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:14:26.962142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:26.962100 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:14:26.968427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:26.968390 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:26.991651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:26.991615 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:14:27.000533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000483 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.000740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000538 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7s8\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.000740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.000740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000691 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.000970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.000970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000834 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.001103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.000974 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.001103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.001006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.001103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.001039 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.102844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.102807 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.102862 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.102960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103023 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103144 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103176 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103206 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103348 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103386 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7s8\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103395 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103445 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103701 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.103803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.103718 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.105451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.105423 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.105586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.105568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.130789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.130743 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.130980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.130808 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7s8\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.297985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.290865 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:27.992415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.992383 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:14:27.994204 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:14:27.994174 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e84ca7b_3a73_4053_9b71_702d4215a8b9.slice/crio-2dd4942e949613d3a64a431a40d63571d5ee0702f8bd8547d17c42061f8482c2 WatchSource:0}: Error finding container 2dd4942e949613d3a64a431a40d63571d5ee0702f8bd8547d17c42061f8482c2: Status 404 returned error can't find the container with id 2dd4942e949613d3a64a431a40d63571d5ee0702f8bd8547d17c42061f8482c2 Oct 29 22:14:27.997036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.997003 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:27.997135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.997073 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:27.997135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:27.997102 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:28.864931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:28.864736 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" event={"ID":"3e84ca7b-3a73-4053-9b71-702d4215a8b9","Type":"ContainerStarted","Data":"ca8a8a534c3816d4e2a506725e6355a880f52a14daa5781a7d26ae390e1550a2"} Oct 29 22:14:28.864931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:28.864806 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" event={"ID":"3e84ca7b-3a73-4053-9b71-702d4215a8b9","Type":"ContainerStarted","Data":"2dd4942e949613d3a64a431a40d63571d5ee0702f8bd8547d17c42061f8482c2"} Oct 29 22:14:28.975667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:28.975606 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podStartSLOduration=2.9755891759999997 podStartE2EDuration="2.975589176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.959235881 +0000 UTC m=+1927.807712728" watchObservedRunningTime="2026-10-29 22:14:28.975589176 +0000 UTC m=+1927.824066021" Oct 29 22:14:29.325916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:29.325877 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:29.327241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:29.327211 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:29.327344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:29.327263 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:30.326281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:30.326247 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:30.326701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:30.326306 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:31.325964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:31.325927 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:31.326150 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:31.325979 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:31.722583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:31.722555 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:14:32.326239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:32.326202 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:32.326450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:32.326258 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:33.325606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:33.325568 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:33.326009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:33.325625 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:34.326650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:34.326563 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:14:34.326650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:34.326622 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:14:35.625633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:35.625596 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:35.626194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:35.626166 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:35.627328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:35.627306 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:14:41.832125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.832089 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:14:41.838035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.838005 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:41.858404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.858367 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-4afaba-serving-cert\"" Oct 29 22:14:41.860433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.860400 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:14:41.934045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.934009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:41.934420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.934388 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:41.934585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.934537 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:41.934848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.934806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8gz\" (UniqueName: \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:41.952985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.952956 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:14:41.957873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.957846 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:41.982262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:41.982222 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071237 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071473 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071532 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071663 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071702 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8gz\" (UniqueName: \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071792 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071857 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.071897 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.074941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.072222 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.081113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.081071 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.087986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.087877 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.088284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.088254 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.088473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.088451 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ljj\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.112499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.104243 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.112499 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:42.104377 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-serving-cert: secret "maas-api-e2e-shared-a-w0-4afaba-serving-cert" not found Oct 29 22:14:42.112499 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:42.104447 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls podName:c239f9a9-c47e-440c-874f-ad512120b4f2 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:42.604428142 +0000 UTC m=+1941.452904981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls") pod "maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" (UID: "c239f9a9-c47e-440c-874f-ad512120b4f2") : secret "maas-api-e2e-shared-a-w0-4afaba-serving-cert" not found Oct 29 22:14:42.150384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.150335 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.152654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.152609 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8gz\" (UniqueName: \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.190745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190705 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.190964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190813 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.190964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190856 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.190964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190885 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.190964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190916 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.190964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190950 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.190979 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ljj\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191146 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191160 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191230 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191289 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191229 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191528 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.191884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.191866 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.193451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.193431 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.193515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.193487 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.217202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.217177 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ljj\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.217359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.217301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.289767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.289731 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:42.615203 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:14:42.615172 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39390aa_e691_498c_971b_2e928c2a2f5e.slice/crio-0258e0dc07607f4871b8bfde04bf34c7a3edc7055de33a0b7eb373c40b52f62c WatchSource:0}: Error finding container 0258e0dc07607f4871b8bfde04bf34c7a3edc7055de33a0b7eb373c40b52f62c: Status 404 returned error can't find the container with id 0258e0dc07607f4871b8bfde04bf34c7a3edc7055de33a0b7eb373c40b52f62c Oct 29 22:14:42.615589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.615565 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:14:42.617932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.617882 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:42.618008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.617966 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:42.618048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.618011 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:14:42.699006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.698564 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.702138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.701986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:42.767371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:42.763997 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:43.166432 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.166337 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" event={"ID":"d39390aa-e691-498c-971b-2e928c2a2f5e","Type":"ContainerStarted","Data":"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a"} Oct 29 22:14:43.170946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.170895 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" event={"ID":"d39390aa-e691-498c-971b-2e928c2a2f5e","Type":"ContainerStarted","Data":"0258e0dc07607f4871b8bfde04bf34c7a3edc7055de33a0b7eb373c40b52f62c"} Oct 29 22:14:43.181799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.181758 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:14:43.231933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.221687 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podStartSLOduration=2.22166692 podStartE2EDuration="2.22166692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:43.220459313 +0000 UTC m=+1942.068936181" watchObservedRunningTime="2026-10-29 22:14:43.22166692 +0000 UTC m=+1942.070143775" Oct 29 22:14:43.290654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.290619 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:43.292371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.292345 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:43.292529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:43.292399 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:44.176787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.176738 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" event={"ID":"c239f9a9-c47e-440c-874f-ad512120b4f2","Type":"ContainerStarted","Data":"c96f6a1e529b1a366a876cab64d13247962dd3076743edb027a318470352194d"} Oct 29 22:14:44.177247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.176984 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:44.177247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.177007 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" event={"ID":"c239f9a9-c47e-440c-874f-ad512120b4f2","Type":"ContainerStarted","Data":"01fd1c006127036aab281138cae74bf90cf7f6bc006efc58664f77107b0c3600"} Oct 29 22:14:44.226080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.226025 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" podStartSLOduration=2.8883789269999998 podStartE2EDuration="3.226010756s" totalImagesPullingTime="337.631829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:44.223184133 +0000 UTC m=+1943.071661007" watchObservedRunningTime="2026-10-29 22:14:44.226010756 +0000 UTC m=+1943.074487602" Oct 29 22:14:44.290704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.290667 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:44.290867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:44.290720 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:45.291251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:45.291218 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:45.291643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:45.291279 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:46.293928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:46.290301 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:46.293928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:46.290364 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:46.691195 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:14:46.691098 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:14:47.290864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:47.290826 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:47.291117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:47.290896 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:48.290145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:48.290106 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:48.290574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:48.290159 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:49.290637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:49.290601 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:49.291083 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:49.290658 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:50.193496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:50.193462 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:14:50.290459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:50.290428 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:14:50.290674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:50.290483 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 29 22:14:52.332063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:52.332028 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:14:52.332498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:52.332103 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:14:52.616409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:52.616321 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:52.616860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:52.616834 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:14:52.617799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:14:52.617777 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:15:00.346489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.346451 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:15:00.352800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.352767 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.375993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.375958 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:15:00.381710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.381480 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0a3d53-serving-cert\"" Oct 29 22:15:00.439277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.439223 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4nqf\" (UniqueName: \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.439277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.439277 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.439659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.439435 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.439659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.439489 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.541383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.541335 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4nqf\" (UniqueName: \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.541603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.541391 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.541603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.541539 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.541736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.541624 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.542351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.542326 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.544115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.544090 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.544115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.544109 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.568100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.568063 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4nqf\" (UniqueName: \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\") pod \"maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:00.676376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:00.676249 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:01.058997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:01.058964 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:15:01.059555 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:15:01.059532 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d385160_2163_4699_ae81_0755966873c5.slice/crio-f71ce09f95d6b58be4bb27169a4b3e37c065ccf5347197f4fbcefdbbd6f296dc WatchSource:0}: Error finding container f71ce09f95d6b58be4bb27169a4b3e37c065ccf5347197f4fbcefdbbd6f296dc: Status 404 returned error can't find the container with id f71ce09f95d6b58be4bb27169a4b3e37c065ccf5347197f4fbcefdbbd6f296dc Oct 29 22:15:01.461189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:01.461114 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" event={"ID":"0d385160-2163-4699-ae81-0755966873c5","Type":"ContainerStarted","Data":"f71ce09f95d6b58be4bb27169a4b3e37c065ccf5347197f4fbcefdbbd6f296dc"} Oct 29 22:15:02.471440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:02.471404 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" event={"ID":"0d385160-2163-4699-ae81-0755966873c5","Type":"ContainerStarted","Data":"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f"} Oct 29 22:15:02.471986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:02.471548 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:02.551928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:02.546232 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" podStartSLOduration=2.079580687 podStartE2EDuration="2.546206805s" totalImagesPullingTime="466.626118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.538563877 +0000 UTC m=+1961.387040724" watchObservedRunningTime="2026-10-29 22:15:02.546206805 +0000 UTC m=+1961.394683652" Oct 29 22:15:08.487199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:08.487168 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:15:16.511237 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:15:16.508633 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:15:22.730981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.727593 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:15:22.738138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.738042 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.777972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.775957 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bef83b\"/\"kube-root-ca.crt\"" Oct 29 22:15:22.782937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.780092 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bef83b\"/\"auth-test-model-bef83b-kserve-self-signed-certs\"" Oct 29 22:15:22.782937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.780160 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bef83b\"/\"openshift-service-ca.crt\"" Oct 29 22:15:22.782937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.780210 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bef83b\"/\"default-dockercfg-7tfvx\"" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.881452 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.881644 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.881764 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.881883 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.882013 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqlj\" (UniqueName: \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.902287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.882114 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.945676 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.945637 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:15:22.984344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.984253 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.985105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.985081 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.985461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.985445 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.985658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.985637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqlj\" (UniqueName: \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.986194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.985853 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.986649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.986625 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.988091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.987551 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.988091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.987609 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.988091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.988044 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.989994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.989023 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:22.989994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:22.989568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:23.019279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.019241 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqlj\" (UniqueName: \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\") pod \"auth-test-model-bef83b-kserve-75b6ddf845-kb659\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:23.083245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.074126 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:15:23.083245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.074197 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:15:23.266979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.260193 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:23.631747 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:15:23.631713 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05e1d894_35f0_419e_b2e6_cca6624024d8.slice/crio-87bff84bb67c5ec43dc957abf8bfd614cd6f3b70b5d1cd1c7edb93803f80d1fb WatchSource:0}: Error finding container 87bff84bb67c5ec43dc957abf8bfd614cd6f3b70b5d1cd1c7edb93803f80d1fb: Status 404 returned error can't find the container with id 87bff84bb67c5ec43dc957abf8bfd614cd6f3b70b5d1cd1c7edb93803f80d1fb Oct 29 22:15:23.633875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.633842 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:15:23.857301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.857254 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" event={"ID":"05e1d894-35f0-419e-b2e6-cca6624024d8","Type":"ContainerStarted","Data":"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592"} Oct 29 22:15:23.857844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.857312 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" event={"ID":"05e1d894-35f0-419e-b2e6-cca6624024d8","Type":"ContainerStarted","Data":"87bff84bb67c5ec43dc957abf8bfd614cd6f3b70b5d1cd1c7edb93803f80d1fb"} Oct 29 22:15:23.910542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:23.910443 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" podStartSLOduration=1.910426075 podStartE2EDuration="1.910426075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:23.902897652 +0000 UTC m=+1982.751374510" watchObservedRunningTime="2026-10-29 22:15:23.910426075 +0000 UTC m=+1982.758902921" Oct 29 22:15:24.272726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:24.272690 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:24.318116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:24.318073 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:15:24.318354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:24.318145 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:15:27.895298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.895261 4376 generic.go:356] "Generic (PLEG): container finished" podID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerID="247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c" exitCode=0 Oct 29 22:15:27.895884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.895318 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerDied","Data":"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c"} Oct 29 22:15:27.896032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.895942 4376 scope.go:122] "RemoveContainer" containerID="247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c" Oct 29 22:15:27.897232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.897211 4376 generic.go:356] "Generic (PLEG): container finished" podID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerID="cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49" exitCode=0 Oct 29 22:15:27.897359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.897334 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerDied","Data":"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49"} Oct 29 22:15:27.897724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.897709 4376 scope.go:122] "RemoveContainer" containerID="cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49" Oct 29 22:15:27.898126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.898091 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:27.898217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.898186 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:27.898287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.898241 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:27.899461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.899432 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:27.899544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.899496 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:27.899544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:27.899531 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:15:28.976516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:28.975794 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerStarted","Data":"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f"} Oct 29 22:15:29.038503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.038366 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerStarted","Data":"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3"} Oct 29 22:15:29.236298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.236215 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:15:29.236494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.236469 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:29.236594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.236532 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:29.310019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.309991 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:15:29.310260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.310237 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:29.310324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:29.310281 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:30.250948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:30.245332 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:30.250948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:30.245399 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:30.318992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:30.314443 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:30.318992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:30.314515 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:31.240931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:31.237320 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:31.240931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:31.237384 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:31.309609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:31.309573 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:31.310049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:31.309631 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:32.237934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:32.237301 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:32.237934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:32.237368 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:32.310319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:32.310277 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:32.310750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:32.310353 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:33.237394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.237350 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:33.237723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.237690 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:33.315439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.315404 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:33.317172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.316056 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:33.317396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.317369 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:33.525692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:33.525607 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:34.132544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:34.132517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:15:34.280948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:34.278471 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:34.280948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:34.278568 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:34.312105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:34.312044 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:34.312461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:34.312422 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:35.237326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:35.237285 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:15:35.237989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:35.237954 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:15:35.310237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:35.310206 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 29 22:15:35.310490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:35.310465 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 29 22:15:36.241510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.241477 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:15:36.242017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.241995 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:15:36.242622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.242599 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:15:36.314585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.314552 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:15:36.314847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.314823 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:15:36.315580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:15:36.315561 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:15:46.435298 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:15:46.435260 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:16:00.031121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.031084 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:16:00.035319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.035290 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.060184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.060157 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:16:00.067405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.067375 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f52917-serving-cert\"" Oct 29 22:16:00.148879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.148847 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.149109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.148895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.149272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.149250 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.149341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.149292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmvd\" (UniqueName: \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.272936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.272450 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.272936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.272526 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmvd\" (UniqueName: \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.272936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.272770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.272936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.272817 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.279697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.279663 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.283080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.283013 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.289399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.289355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.310797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.310761 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmvd\" (UniqueName: \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\") pod \"maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.359405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.359222 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:00.741362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.741332 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:16:00.741655 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:16:00.741633 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b2c47_75c9_4ad7_9090_a2e1a1320fd4.slice/crio-97e2a985008509d84428a24bfa95b6b32f65b3bd72f7579257ac7f1458607476 WatchSource:0}: Error finding container 97e2a985008509d84428a24bfa95b6b32f65b3bd72f7579257ac7f1458607476: Status 404 returned error can't find the container with id 97e2a985008509d84428a24bfa95b6b32f65b3bd72f7579257ac7f1458607476 Oct 29 22:16:00.743170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:00.743151 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:16:01.631867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:01.631829 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" event={"ID":"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4","Type":"ContainerStarted","Data":"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f"} Oct 29 22:16:01.632364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:01.631951 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" event={"ID":"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4","Type":"ContainerStarted","Data":"97e2a985008509d84428a24bfa95b6b32f65b3bd72f7579257ac7f1458607476"} Oct 29 22:16:01.632364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:01.632046 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:01.689181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:01.689125 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" podStartSLOduration=1.2776508469999999 podStartE2EDuration="1.689107693s" totalImagesPullingTime="411.456846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:01.688198854 +0000 UTC m=+2020.536675704" watchObservedRunningTime="2026-10-29 22:16:01.689107693 +0000 UTC m=+2020.537584540" Oct 29 22:16:07.689266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:07.683130 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:16:16.559821 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:16:16.559780 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:16:20.878497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.878462 4376 generic.go:356] "Generic (PLEG): container finished" podID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerID="fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e" exitCode=0 Oct 29 22:16:20.878954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.878535 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerDied","Data":"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e"} Oct 29 22:16:20.879004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.878949 4376 scope.go:122] "RemoveContainer" containerID="fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e" Oct 29 22:16:20.880744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.880710 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:20.880828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.880792 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:20.880867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:20.880829 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:21.889055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:21.889012 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerStarted","Data":"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826"} Oct 29 22:16:21.919613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:21.919582 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:16:21.921312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:21.921284 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:21.921437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:21.921361 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:22.919134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:22.919102 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:22.919543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:22.919155 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:23.922258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:23.922216 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:23.922930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:23.922882 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:24.757063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:24.757023 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:16:24.919972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:24.919939 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:24.920138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:24.919999 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:25.919943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:25.919879 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:25.920375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:25.919966 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:26.919222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:26.919186 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:16:26.919406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:26.919255 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:16:27.932074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:27.932044 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:16:27.933342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:27.933318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:16:30.682245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.682214 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:16:30.686393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.686368 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.711664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.711622 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:16:30.729833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.729801 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bef83b-openshift-default-dockercfg-hrrzn\"" Oct 29 22:16:30.792366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.774104 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803091 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803350 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhvm\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803494 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803614 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.803735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.803719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.817964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.809821 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.817964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.809922 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.911974 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912035 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912159 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912222 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhvm\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912277 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912301 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912359 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912439 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912471 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912578 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.912843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.912815 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.913105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.913083 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.914722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.914700 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.914945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.914896 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.937975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.937874 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:30.938797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:30.938769 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhvm\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:31.000567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.000535 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:31.325835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.325798 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:16:31.326396 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:16:31.326361 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2b1360_5739_4485_b57e_00aa770854a3.slice/crio-f3c3251ef90d8b107509e3b58759a688398f12365c6bf2076f8d0cf6ae634e6a WatchSource:0}: Error finding container f3c3251ef90d8b107509e3b58759a688398f12365c6bf2076f8d0cf6ae634e6a: Status 404 returned error can't find the container with id f3c3251ef90d8b107509e3b58759a688398f12365c6bf2076f8d0cf6ae634e6a Oct 29 22:16:31.329586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.329543 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:31.329697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.329618 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:31.329697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.329649 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:16:31.978016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.977473 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" event={"ID":"5f2b1360-5739-4485-b57e-00aa770854a3","Type":"ContainerStarted","Data":"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381"} Oct 29 22:16:31.978016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:31.977525 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" event={"ID":"5f2b1360-5739-4485-b57e-00aa770854a3","Type":"ContainerStarted","Data":"f3c3251ef90d8b107509e3b58759a688398f12365c6bf2076f8d0cf6ae634e6a"} Oct 29 22:16:32.000987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:32.000952 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:32.002077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:32.002056 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:32.002203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:32.002099 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:32.075765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:32.075705 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podStartSLOduration=2.075688514 podStartE2EDuration="2.075688514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:32.057861473 +0000 UTC m=+2050.906338320" watchObservedRunningTime="2026-10-29 22:16:32.075688514 +0000 UTC m=+2050.924165420" Oct 29 22:16:33.001627 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:33.001594 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:33.002072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:33.001642 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:34.005319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:34.005272 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:34.005835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:34.005347 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:35.001108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:35.001070 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:35.001285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:35.001128 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:36.001161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:36.001124 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:36.001571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:36.001183 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:37.001297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:37.001265 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:16:37.001687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:37.001330 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 29 22:16:38.006557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:38.006521 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:38.007048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:38.006831 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:38.007564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:16:38.007542 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:16:46.436628 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:16:46.436591 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:17:07.881117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:07.881081 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:17:07.885495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:07.885471 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:07.914701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:07.914661 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:17:07.917492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:07.917464 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f52917-openshift-default-dockercfg-2q4hk\"" Oct 29 22:17:08.012845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.012806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.012853 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013158 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcfd\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013246 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013272 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013546 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013531 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.013616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.013560 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.114861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.114828 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcfd\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.114915 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.114933 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.114956 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115109 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115129 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115155 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115177 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115235 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115319 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115441 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115561 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115690 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.115993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.115972 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.117477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.117456 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.117801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.117778 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.141457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.141389 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.141611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.141591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcfd\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.203587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.203553 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:08.538570 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:17:08.538524 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd93542_5857_4474_bc8d_63f794f871cf.slice/crio-f2b8d3cefcbc8dec8067c8288861cea746a17fd43c297f2543d29a31fa6a2b3c WatchSource:0}: Error finding container f2b8d3cefcbc8dec8067c8288861cea746a17fd43c297f2543d29a31fa6a2b3c: Status 404 returned error can't find the container with id f2b8d3cefcbc8dec8067c8288861cea746a17fd43c297f2543d29a31fa6a2b3c Oct 29 22:17:08.538702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.538633 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:17:08.541708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.541671 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:08.541803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.541748 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:08.541803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:08.541778 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:09.438662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:09.438613 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" event={"ID":"2cd93542-5857-4474-bc8d-63f794f871cf","Type":"ContainerStarted","Data":"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3"} Oct 29 22:17:09.440979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:09.440948 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" event={"ID":"2cd93542-5857-4474-bc8d-63f794f871cf","Type":"ContainerStarted","Data":"f2b8d3cefcbc8dec8067c8288861cea746a17fd43c297f2543d29a31fa6a2b3c"} Oct 29 22:17:09.525862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:09.525809 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podStartSLOduration=2.525794732 podStartE2EDuration="2.525794732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:09.507797352 +0000 UTC m=+2088.356274199" watchObservedRunningTime="2026-10-29 22:17:09.525794732 +0000 UTC m=+2088.374271577" Oct 29 22:17:10.203937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:10.203893 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:10.205497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:10.205469 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.205637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:10.205522 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 29 22:17:11.204472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:11.204436 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.204854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:11.204498 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 29 22:17:12.204830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:12.204794 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.205402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:12.204866 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 29 22:17:13.204506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:13.204471 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.204692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:13.204534 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 29 22:17:14.204401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:14.204370 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.204880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:14.204422 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 29 22:17:15.209665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:15.209636 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:15.210148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:15.210129 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:15.210748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:15.210728 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:17:16.445130 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:17:16.445093 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:17:46.436254 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:17:46.436167 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:17:53.020778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.020742 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:17:53.024619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.024598 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.075751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.075716 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:17:53.118832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.118802 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.118860 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119003 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119076 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119215 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119261 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119355 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119472 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqt2\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.119523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.119504 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.220792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.220756 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.220995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.220818 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.220995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.220868 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221083 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.220990 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221107 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221157 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221233 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221359 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221375 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221415 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqt2\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221470 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221693 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.221974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.221953 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.223376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.223350 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.223563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.223541 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.245619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.245591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.245713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.245685 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqt2\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.341367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.341297 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:53.670226 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:17:53.670195 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37000434_a758_4adc_8948_b541123008da.slice/crio-02ed127b757f1f2fa1026d6a026fc16414a11342eab0febf7cc1b3491998d1f1 WatchSource:0}: Error finding container 02ed127b757f1f2fa1026d6a026fc16414a11342eab0febf7cc1b3491998d1f1: Status 404 returned error can't find the container with id 02ed127b757f1f2fa1026d6a026fc16414a11342eab0febf7cc1b3491998d1f1 Oct 29 22:17:53.670848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.670824 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:17:53.673550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.673517 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:53.673632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.673581 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:53.673632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:53.673612 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:17:54.107507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.107460 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" event={"ID":"37000434-a758-4adc-8948-b541123008da","Type":"ContainerStarted","Data":"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea"} Oct 29 22:17:54.108034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.107539 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" event={"ID":"37000434-a758-4adc-8948-b541123008da","Type":"ContainerStarted","Data":"02ed127b757f1f2fa1026d6a026fc16414a11342eab0febf7cc1b3491998d1f1"} Oct 29 22:17:54.160521 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.160455 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podStartSLOduration=2.160436886 podStartE2EDuration="2.160436886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:54.15415328 +0000 UTC m=+2133.002630125" watchObservedRunningTime="2026-10-29 22:17:54.160436886 +0000 UTC m=+2133.008913732" Oct 29 22:17:54.342060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.342031 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:17:54.343875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.343849 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:54.344040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:54.343935 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:17:55.358320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:55.358278 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:55.358728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:55.358341 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:17:56.353452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:56.353414 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:56.353733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:56.353704 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:17:57.348934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:57.348307 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:57.348934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:57.348376 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:17:58.343928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:58.343672 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:58.343928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:58.343773 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:17:59.342927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:59.342287 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:17:59.342927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:17:59.342347 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:18:00.347737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:00.347701 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:18:00.348234 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:00.348193 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:18:00.349124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:00.349097 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:18:13.798653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:13.798614 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:18:13.800178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:13.799897 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" containerID="cri-o://84e62862974aa5b4c09906a81496833c2dabba7441e702c19bde2c1ae59144f8" gracePeriod=30 Oct 29 22:18:16.435298 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:18:16.435252 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:18:18.976649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:18.976613 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 22:18:18.977182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:18.976834 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" containerID="cri-o://54d1d90d7b181bedf87e78ac216c4410d099e13a0f7fe18b2d47ed2dec88f8ed" gracePeriod=30 Oct 29 22:18:19.574557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:19.574522 4376 generic.go:356] "Generic (PLEG): container finished" podID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerID="84e62862974aa5b4c09906a81496833c2dabba7441e702c19bde2c1ae59144f8" exitCode=0 Oct 29 22:18:19.574748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:19.574591 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" event={"ID":"19d7191d-4570-4090-8589-8e73dd50f7c6","Type":"ContainerDied","Data":"84e62862974aa5b4c09906a81496833c2dabba7441e702c19bde2c1ae59144f8"} Oct 29 22:18:19.880074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:19.866651 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107163 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107254 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107342 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107387 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107461 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107493 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107578 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.107636 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.110645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.108081 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\") pod \"19d7191d-4570-4090-8589-8e73dd50f7c6\" (UID: \"19d7191d-4570-4090-8589-8e73dd50f7c6\") " Oct 29 22:18:20.114005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.113442 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:20.117226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.116894 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:20.118839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.118760 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:20.119057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.119015 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:20.119231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.119170 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:18:20.126229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.126203 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "kube-api-access-d7rs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:20.201671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.200774 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:18:20.201671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.200777 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:20.208097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.208046 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token" pod "19d7191d-4570-4090-8589-8e73dd50f7c6" (UID: "19d7191d-4570-4090-8589-8e73dd50f7c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:20.214094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214062 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7rs5\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-kube-api-access-d7rs5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214097 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d7191d-4570-4090-8589-8e73dd50f7c6-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214114 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214128 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214142 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214155 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214169 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214182 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.214219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.214196 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d7191d-4570-4090-8589-8e73dd50f7c6-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:20.584874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.584842 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" Oct 29 22:18:20.584874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.584852 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9" event={"ID":"19d7191d-4570-4090-8589-8e73dd50f7c6","Type":"ContainerDied","Data":"bd2a1ddda8c00c97da2f94b5cd88451ce97f6afc7623cb82a30d8fc942ce1fc0"} Oct 29 22:18:20.585166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.584941 4376 scope.go:122] "RemoveContainer" containerID="84e62862974aa5b4c09906a81496833c2dabba7441e702c19bde2c1ae59144f8" Oct 29 22:18:20.640431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.640398 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:18:20.641378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.641354 4376 cpu_manager.go:414] "containerMap: removing container" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" Oct 29 22:18:20.641378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.641376 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" Oct 29 22:18:20.641637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.641621 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" containerName="istio-proxy" Oct 29 22:18:20.645767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.645745 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.647166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.647147 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:18:20.661269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.661238 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wn8q9"] Oct 29 22:18:20.663524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.663484 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0c3d9a-serving-cert\"" Oct 29 22:18:20.667138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.667116 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:18:20.824624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.824592 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.824797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.824752 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.824885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.824863 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rns9\" (UniqueName: \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.825022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.824994 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.930973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.926752 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.930973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.926996 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.930973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.927101 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rns9\" (UniqueName: \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.930973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.927172 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.930973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.928507 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.973533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.972493 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.977892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.977814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rns9\" (UniqueName: \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:20.978455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:20.978429 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:21.270863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:21.270831 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:21.276720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:21.276696 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d7191d-4570-4090-8589-8e73dd50f7c6" path="/var/lib/kubelet/pods/19d7191d-4570-4090-8589-8e73dd50f7c6/volumes" Oct 29 22:18:21.626103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:21.626075 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:18:21.626704 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:18:21.626678 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda401e66c_872c_4d84_bf69_213932b8cd09.slice/crio-a4b6e716a18d47eb956e178358717a424062b678975a69cb99c8f2d95c8b1be4 WatchSource:0}: Error finding container a4b6e716a18d47eb956e178358717a424062b678975a69cb99c8f2d95c8b1be4: Status 404 returned error can't find the container with id a4b6e716a18d47eb956e178358717a424062b678975a69cb99c8f2d95c8b1be4 Oct 29 22:18:22.641820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:22.639600 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" event={"ID":"a401e66c-872c-4d84-bf69-213932b8cd09","Type":"ContainerStarted","Data":"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809"} Oct 29 22:18:22.641820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:22.639678 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" event={"ID":"a401e66c-872c-4d84-bf69-213932b8cd09","Type":"ContainerStarted","Data":"a4b6e716a18d47eb956e178358717a424062b678975a69cb99c8f2d95c8b1be4"} Oct 29 22:18:22.641820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:22.641309 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:22.687948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:22.687872 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" podStartSLOduration=2.304154923 podStartE2EDuration="2.687858435s" totalImagesPullingTime="383.703512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:22.685820979 +0000 UTC m=+2161.534297824" watchObservedRunningTime="2026-10-29 22:18:22.687858435 +0000 UTC m=+2161.536335280" Oct 29 22:18:24.291679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.291654 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361247 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361391 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361431 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361472 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361508 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361577 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361638 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.369981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.361662 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\") pod \"f0d57a57-ac8c-43f8-a235-5b8952fba093\" (UID: \"f0d57a57-ac8c-43f8-a235-5b8952fba093\") " Oct 29 22:18:24.372507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.372266 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:18:24.375926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.372704 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:24.397928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.376123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:24.397928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.378401 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:24.397928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.396558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:18:24.398518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.398460 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:24.407756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.406400 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "kube-api-access-57gwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:24.407756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.406957 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:24.412062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.410098 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token" pod "f0d57a57-ac8c-43f8-a235-5b8952fba093" (UID: "f0d57a57-ac8c-43f8-a235-5b8952fba093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480050 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57gwl\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-kube-api-access-57gwl\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480084 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480097 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d57a57-ac8c-43f8-a235-5b8952fba093-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480109 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480120 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480131 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480141 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d57a57-ac8c-43f8-a235-5b8952fba093-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480152 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.480619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.480162 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d57a57-ac8c-43f8-a235-5b8952fba093-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:24.588162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.588084 4376 scope.go:122] "RemoveContainer" containerID="54d1d90d7b181bedf87e78ac216c4410d099e13a0f7fe18b2d47ed2dec88f8ed" Oct 29 22:18:24.655109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.655066 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" event={"ID":"f0d57a57-ac8c-43f8-a235-5b8952fba093","Type":"ContainerDied","Data":"54d1d90d7b181bedf87e78ac216c4410d099e13a0f7fe18b2d47ed2dec88f8ed"} Oct 29 22:18:24.655304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.655133 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" event={"ID":"f0d57a57-ac8c-43f8-a235-5b8952fba093","Type":"ContainerDied","Data":"b557d0cb4a5cdd55aa3215ea0f0e7019b44cabb5204e6e2c70d0b74de101a06b"} Oct 29 22:18:24.655304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.655145 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l" Oct 29 22:18:24.743135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.743106 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 22:18:24.750525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:24.750498 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mmg4l"] Oct 29 22:18:25.274401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:25.274359 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" path="/var/lib/kubelet/pods/f0d57a57-ac8c-43f8-a235-5b8952fba093/volumes" Oct 29 22:18:29.660540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:29.660512 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:18:46.437441 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:18:46.437400 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:18:56.401205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.401160 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:18:56.401990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.401972 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" Oct 29 22:18:56.401990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.401989 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" Oct 29 22:18:56.402138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.402115 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d57a57-ac8c-43f8-a235-5b8952fba093" containerName="istio-proxy" Oct 29 22:18:56.406389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.406361 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.432236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.432203 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:18:56.439053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.439025 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7d8760-serving-cert\"" Oct 29 22:18:56.509292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.509252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dh9m\" (UniqueName: \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.509454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.509407 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.509531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.509516 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.509567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.509542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.611089 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.611248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.611277 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.611497 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dh9m\" (UniqueName: \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.612628 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.617817 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:18:56.625029 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-serving-cert: secret "maas-api-e2e-extmod-b-7d8760-serving-cert" not found Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:18:56.625133 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls podName:c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9 nodeName:}" failed. No retries permitted until 2026-10-29 22:18:57.125111848 +0000 UTC m=+2195.973588671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls") pod "maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" (UID: "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9") : secret "maas-api-e2e-extmod-b-7d8760-serving-cert" not found Oct 29 22:18:56.642951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:56.640440 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dh9m\" (UniqueName: \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:57.221654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:57.221612 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:57.224050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:57.224020 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:57.345726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:57.345693 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:57.798354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:57.798322 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:18:57.798725 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:18:57.798615 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bc3b9e_325c_43c6_9211_0ad5d8fe87c9.slice/crio-94529efdda21b9092d044861cc034f33c7cee854b17cb6c318785df9c4a2bc68 WatchSource:0}: Error finding container 94529efdda21b9092d044861cc034f33c7cee854b17cb6c318785df9c4a2bc68: Status 404 returned error can't find the container with id 94529efdda21b9092d044861cc034f33c7cee854b17cb6c318785df9c4a2bc68 Oct 29 22:18:58.157031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:58.156930 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" event={"ID":"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9","Type":"ContainerStarted","Data":"94529efdda21b9092d044861cc034f33c7cee854b17cb6c318785df9c4a2bc68"} Oct 29 22:18:59.166408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:59.166375 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" event={"ID":"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9","Type":"ContainerStarted","Data":"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8"} Oct 29 22:18:59.166869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:59.166452 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:18:59.214168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:18:59.214118 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" podStartSLOduration=2.8201915509999997 podStartE2EDuration="3.214104551s" totalImagesPullingTime="393.913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:59.209257736 +0000 UTC m=+2198.057734618" watchObservedRunningTime="2026-10-29 22:18:59.214104551 +0000 UTC m=+2198.062581395" Oct 29 22:19:04.175324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.167144 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 22:19:04.178965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.178252 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:19:04.178965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.178509 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" containerID="cri-o://296218c5be44c24eecdceab37420e4aa4cfdb4af2a2060bfb408ac10d9379632" gracePeriod=30 Oct 29 22:19:04.188959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.188927 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 22:19:04.189373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.189340 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" containerID="cri-o://e8b15e48f160cdc34e35fc7d8dc1ee16717ff0b3b5c876f5d8dd9b1e14d4cc5d" gracePeriod=30 Oct 29 22:19:04.398352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:04.398307 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" containerID="cri-o://939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8" gracePeriod=30 Oct 29 22:19:05.178391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:05.178356 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:19:09.441213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.441165 4376 generic.go:356] "Generic (PLEG): container finished" podID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerID="296218c5be44c24eecdceab37420e4aa4cfdb4af2a2060bfb408ac10d9379632" exitCode=0 Oct 29 22:19:09.441557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.441273 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" event={"ID":"7275c8f6-719f-43b4-b7e9-a50b89309a51","Type":"ContainerDied","Data":"296218c5be44c24eecdceab37420e4aa4cfdb4af2a2060bfb408ac10d9379632"} Oct 29 22:19:09.443637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.443557 4376 generic.go:356] "Generic (PLEG): container finished" podID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerID="e8b15e48f160cdc34e35fc7d8dc1ee16717ff0b3b5c876f5d8dd9b1e14d4cc5d" exitCode=0 Oct 29 22:19:09.443637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.443625 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" event={"ID":"43a36fd4-d33d-4038-8436-ca2a12b7fa25","Type":"ContainerDied","Data":"e8b15e48f160cdc34e35fc7d8dc1ee16717ff0b3b5c876f5d8dd9b1e14d4cc5d"} Oct 29 22:19:09.593979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.593954 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:19:09.605440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.605416 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 22:19:09.687864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.687828 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.687882 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.687947 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.687979 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688008 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688063 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688114 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688158 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688224 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688223 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688319 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688334 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688410 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688565 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688628 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688569 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.688664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688624 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688700 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688763 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\") pod \"7275c8f6-719f-43b4-b7e9-a50b89309a51\" (UID: \"7275c8f6-719f-43b4-b7e9-a50b89309a51\") " Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688812 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688872 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.688991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.688922 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\") pod \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\" (UID: \"43a36fd4-d33d-4038-8436-ca2a12b7fa25\") " Oct 29 22:19:09.689267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689084 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:09.689350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689325 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.689623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689599 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.689846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.689958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689920 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.689958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689942 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.689958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689956 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689970 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689983 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.689997 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7275c8f6-719f-43b4-b7e9-a50b89309a51-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.690010 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.690022 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.690139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.690036 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.690895 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "kube-api-access-bwvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.691821 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.691826 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.691955 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.692187 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "kube-api-access-ddbmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.692302 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token" pod "7275c8f6-719f-43b4-b7e9-a50b89309a51" (UID: "7275c8f6-719f-43b4-b7e9-a50b89309a51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:09.692484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.692475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:09.692968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.692696 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy" pod "43a36fd4-d33d-4038-8436-ca2a12b7fa25" (UID: "43a36fd4-d33d-4038-8436-ca2a12b7fa25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:09.792087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792044 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792077 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvdq\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-kube-api-access-bwvdq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792090 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792101 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7275c8f6-719f-43b4-b7e9-a50b89309a51-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792113 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792123 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792134 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbmx\" (UniqueName: \"kubernetes.io/projected/7275c8f6-719f-43b4-b7e9-a50b89309a51-kube-api-access-ddbmx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792144 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a36fd4-d33d-4038-8436-ca2a12b7fa25-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:09.792348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:09.792155 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a36fd4-d33d-4038-8436-ca2a12b7fa25-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.071209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.071183 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 22:19:10.198599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198561 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.198599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198604 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.198846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198672 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.198846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198706 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.198846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198812 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.199029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198879 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.199029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198974 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:10.199029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.198973 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:10.199029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199010 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.199230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:10.199230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199206 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.199350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199313 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\") pod \"48a6c50c-d11b-4820-b3d2-50af7f22e936\" (UID: \"48a6c50c-d11b-4820-b3d2-50af7f22e936\") " Oct 29 22:19:10.199406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199358 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:10.199491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.199456 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:10.200127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.200097 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a6c50c-d11b-4820-b3d2-50af7f22e936-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.200127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.200120 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.200335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.200133 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.200335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.200146 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.200335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.200158 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.201518 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.201475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:10.201639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.201555 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:10.201639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.201607 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "kube-api-access-5dfh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:10.201761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.201634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo" pod "48a6c50c-d11b-4820-b3d2-50af7f22e936" (UID: "48a6c50c-d11b-4820-b3d2-50af7f22e936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:10.302515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.302476 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.302515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.302509 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dfh5\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-kube-api-access-5dfh5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.302515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.302518 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.302759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.302527 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a6c50c-d11b-4820-b3d2-50af7f22e936-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:10.453349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.453320 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" Oct 29 22:19:10.453784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.453354 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg" event={"ID":"7275c8f6-719f-43b4-b7e9-a50b89309a51","Type":"ContainerDied","Data":"cbc28cb51ab988cf102e09fea80a35db8c58a9ca134c426e161e542a55aff2bd"} Oct 29 22:19:10.453784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.453420 4376 scope.go:122] "RemoveContainer" containerID="296218c5be44c24eecdceab37420e4aa4cfdb4af2a2060bfb408ac10d9379632" Oct 29 22:19:10.455038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.455023 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" Oct 29 22:19:10.455159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.455029 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz" event={"ID":"43a36fd4-d33d-4038-8436-ca2a12b7fa25","Type":"ContainerDied","Data":"1a7220c80904c19fadb6f654773c127ae88f3af88056fcc210ea243251865078"} Oct 29 22:19:10.463963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.460756 4376 generic.go:356] "Generic (PLEG): container finished" podID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerID="939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8" exitCode=0 Oct 29 22:19:10.463963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.460856 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" event={"ID":"48a6c50c-d11b-4820-b3d2-50af7f22e936","Type":"ContainerDied","Data":"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8"} Oct 29 22:19:10.463963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.460967 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" event={"ID":"48a6c50c-d11b-4820-b3d2-50af7f22e936","Type":"ContainerDied","Data":"a13e4a3990cb6794808809333b9eb9ae4fbaac961e8ed0815e5c4ee231c07ed3"} Oct 29 22:19:10.463963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.461347 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz" Oct 29 22:19:10.475688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.475668 4376 scope.go:122] "RemoveContainer" containerID="e8b15e48f160cdc34e35fc7d8dc1ee16717ff0b3b5c876f5d8dd9b1e14d4cc5d" Oct 29 22:19:10.487292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.487269 4376 scope.go:122] "RemoveContainer" containerID="939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8" Oct 29 22:19:10.497191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.497157 4376 scope.go:122] "RemoveContainer" containerID="939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8" Oct 29 22:19:10.497470 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:10.497449 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8\": container with ID starting with 939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8 not found: ID does not exist" containerID="939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8" Oct 29 22:19:10.497522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.497480 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8"} err="failed to get container status \"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8\": rpc error: code = NotFound desc = could not find container \"939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8\": container with ID starting with 939bac31e999ffbd7aeaeb138ae85dad17a235442b58f1ecb12ad0a4e1d9f0d8 not found: ID does not exist" Oct 29 22:19:10.546437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.546402 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:19:10.559332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.559298 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-7l4dg"] Oct 29 22:19:10.590275 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.590244 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 22:19:10.599020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.598987 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-z7brz"] Oct 29 22:19:10.634644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.634570 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 22:19:10.639621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:10.639596 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jzkkz"] Oct 29 22:19:11.102021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.101988 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:19:11.102852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102827 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" Oct 29 22:19:11.102852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102848 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" Oct 29 22:19:11.103088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102861 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" Oct 29 22:19:11.103088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102869 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" Oct 29 22:19:11.103088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102926 4376 cpu_manager.go:414] "containerMap: removing container" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" Oct 29 22:19:11.103088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.102934 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" Oct 29 22:19:11.103710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.103196 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" containerName="istio-proxy" Oct 29 22:19:11.103710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.103227 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" containerName="istio-proxy" Oct 29 22:19:11.103710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.103238 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" containerName="istio-proxy" Oct 29 22:19:11.109690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.109667 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.129962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.126541 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-07e166-openshift-default-dockercfg-nf577\"" Oct 29 22:19:11.132839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.132803 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:19:11.216718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.216684 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.216718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.216722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.216961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.216756 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.216961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.216888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.217141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.217121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.217212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.217192 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.217281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.217266 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmqf\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.217328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.217295 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.217328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.217319 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.275172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.275132 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a36fd4-d33d-4038-8436-ca2a12b7fa25" path="/var/lib/kubelet/pods/43a36fd4-d33d-4038-8436-ca2a12b7fa25/volumes" Oct 29 22:19:11.275730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.275715 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a6c50c-d11b-4820-b3d2-50af7f22e936" path="/var/lib/kubelet/pods/48a6c50c-d11b-4820-b3d2-50af7f22e936/volumes" Oct 29 22:19:11.276218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.276205 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7275c8f6-719f-43b4-b7e9-a50b89309a51" path="/var/lib/kubelet/pods/7275c8f6-719f-43b4-b7e9-a50b89309a51/volumes" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329064 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329147 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329209 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmqf\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329280 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329417 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329445 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329488 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.332074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.329546 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.334796 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.335112 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.335591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.335804 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.337201 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.345509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.345232 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.350937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.346929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.365749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.365679 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmqf\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.380782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.380749 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:11.432250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:11.432211 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:12.000474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.000443 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:19:12.000842 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:19:12.000561 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d2bc07_2df6_4758_b256_995e638df935.slice/crio-3ac7a542a8aeeb61cc52d55d090b9195bcacd47afda69922cabcd9a662f17bb2 WatchSource:0}: Error finding container 3ac7a542a8aeeb61cc52d55d090b9195bcacd47afda69922cabcd9a662f17bb2: Status 404 returned error can't find the container with id 3ac7a542a8aeeb61cc52d55d090b9195bcacd47afda69922cabcd9a662f17bb2 Oct 29 22:19:12.003506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.003470 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:19:12.003623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.003545 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:19:12.003689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.003623 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:19:12.566792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.566753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerStarted","Data":"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1"} Oct 29 22:19:12.567015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:12.566820 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerStarted","Data":"3ac7a542a8aeeb61cc52d55d090b9195bcacd47afda69922cabcd9a662f17bb2"} Oct 29 22:19:13.433052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:13.433015 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:13.439366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:13.439339 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:13.500798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:13.500744 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podStartSLOduration=2.500731162 podStartE2EDuration="2.500731162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:12.666384623 +0000 UTC m=+2211.514861468" watchObservedRunningTime="2026-10-29 22:19:13.500731162 +0000 UTC m=+2212.349208006" Oct 29 22:19:13.576134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:13.576100 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:13.577015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:13.576989 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:19:16.435651 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:16.435600 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:19:21.831673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:21.831636 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:19:21.832126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:21.831848 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" containerID="cri-o://ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f" gracePeriod=30 Oct 29 22:19:22.760396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:22.760360 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:19:22.760612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:22.760584 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" containerID="cri-o://5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd" gracePeriod=30 Oct 29 22:19:26.791372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:26.791323 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 22:19:26.791883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:26.791569 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" containerID="cri-o://e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba" gracePeriod=30 Oct 29 22:19:27.125865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.125835 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:19:27.198132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198100 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198309 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198175 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198309 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198202 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198356 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198409 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198439 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198477 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198567 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:27.198734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198645 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198744 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\") pod \"f89b453d-cce6-4ca8-9133-ab91353f1249\" (UID: \"f89b453d-cce6-4ca8-9133-ab91353f1249\") " Oct 29 22:19:27.198850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:27.198850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.198812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:27.199029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199009 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:27.199088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199066 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:27.199709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199682 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.199809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199716 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.199809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199731 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.199809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199745 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89b453d-cce6-4ca8-9133-ab91353f1249-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.199809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.199758 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.201308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.201276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:27.201401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.201309 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:27.201994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.201972 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:27.202157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.202137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4" pod "f89b453d-cce6-4ca8-9133-ab91353f1249" (UID: "f89b453d-cce6-4ca8-9133-ab91353f1249"). InnerVolumeSpecName "kube-api-access-fpnl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:27.301438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.301395 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.301438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.301436 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.301673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.301449 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89b453d-cce6-4ca8-9133-ab91353f1249-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.301673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.301462 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnl4\" (UniqueName: \"kubernetes.io/projected/f89b453d-cce6-4ca8-9133-ab91353f1249-kube-api-access-fpnl4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:27.683868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.683834 4376 generic.go:356] "Generic (PLEG): container finished" podID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerID="ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f" exitCode=0 Oct 29 22:19:27.684102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.683897 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" Oct 29 22:19:27.684102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.683938 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" event={"ID":"f89b453d-cce6-4ca8-9133-ab91353f1249","Type":"ContainerDied","Data":"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f"} Oct 29 22:19:27.684102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.683998 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn" event={"ID":"f89b453d-cce6-4ca8-9133-ab91353f1249","Type":"ContainerDied","Data":"1d542a58fceeb6029c50dd96ae2766251f9d7ae60703a01a11d85cfdfb91ecd1"} Oct 29 22:19:27.684102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.684024 4376 scope.go:122] "RemoveContainer" containerID="ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f" Oct 29 22:19:27.695078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.695061 4376 scope.go:122] "RemoveContainer" containerID="ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f" Oct 29 22:19:27.695344 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:27.695320 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f\": container with ID starting with ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f not found: ID does not exist" containerID="ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f" Oct 29 22:19:27.695388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.695358 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f"} err="failed to get container status \"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f\": rpc error: code = NotFound desc = could not find container \"ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f\": container with ID starting with ecf02564743896b6dffa1db93f8dfe8db9b4115ddc0ea2b6834e3f5db1d8348f not found: ID does not exist" Oct 29 22:19:27.742693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.742661 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:19:27.754979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:27.754951 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-v5nrn"] Oct 29 22:19:28.050034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.050004 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:19:28.115680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.115644 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.115862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.115707 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116067 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.116206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116077 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116137 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116388 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116433 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116483 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116601 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.116832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.116811 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\") pod \"2081f513-c697-494f-9031-b854d97de08c\" (UID: \"2081f513-c697-494f-9031-b854d97de08c\") " Oct 29 22:19:28.117312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117109 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.117312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117184 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.117312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117250 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:28.117579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117462 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.117708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117686 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.117776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117712 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2081f513-c697-494f-9031-b854d97de08c-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.117776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117725 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.117776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117737 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.117776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.117752 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.118840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.118807 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "kube-api-access-22w22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.119035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.118981 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:28.119311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.119285 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.119617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.119588 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token" pod "2081f513-c697-494f-9031-b854d97de08c" (UID: "2081f513-c697-494f-9031-b854d97de08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.219446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.219397 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22w22\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-kube-api-access-22w22\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.219446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.219441 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2081f513-c697-494f-9031-b854d97de08c-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.219446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.219451 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2081f513-c697-494f-9031-b854d97de08c-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.219446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.219459 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2081f513-c697-494f-9031-b854d97de08c-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.693100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.693015 4376 generic.go:356] "Generic (PLEG): container finished" podID="2081f513-c697-494f-9031-b854d97de08c" containerID="5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd" exitCode=0 Oct 29 22:19:28.693100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.693065 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" Oct 29 22:19:28.693301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.693096 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" event={"ID":"2081f513-c697-494f-9031-b854d97de08c","Type":"ContainerDied","Data":"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd"} Oct 29 22:19:28.693301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.693150 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs" event={"ID":"2081f513-c697-494f-9031-b854d97de08c","Type":"ContainerDied","Data":"56d2adc6910708a1957ec2822adacf1001298222d8a9f4f898bb82b9f2213691"} Oct 29 22:19:28.693301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.693171 4376 scope.go:122] "RemoveContainer" containerID="5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd" Oct 29 22:19:28.720006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.719977 4376 scope.go:122] "RemoveContainer" containerID="5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd" Oct 29 22:19:28.720328 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:28.720302 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd\": container with ID starting with 5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd not found: ID does not exist" containerID="5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd" Oct 29 22:19:28.720405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.720336 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd"} err="failed to get container status \"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd\": rpc error: code = NotFound desc = could not find container \"5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd\": container with ID starting with 5bc4d222363e16c5a9400033bec08e09d16f4cbdef33d5b2df82d1de0efc6bfd not found: ID does not exist" Oct 29 22:19:28.762784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.762750 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:19:28.775467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:28.775435 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-84qzs"] Oct 29 22:19:29.273785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.273753 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2081f513-c697-494f-9031-b854d97de08c" path="/var/lib/kubelet/pods/2081f513-c697-494f-9031-b854d97de08c/volumes" Oct 29 22:19:29.274312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.274296 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" path="/var/lib/kubelet/pods/f89b453d-cce6-4ca8-9133-ab91353f1249/volumes" Oct 29 22:19:29.851478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.851445 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:19:29.852251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852233 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" Oct 29 22:19:29.852251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852249 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" Oct 29 22:19:29.852377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852303 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" Oct 29 22:19:29.852377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852309 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" Oct 29 22:19:29.852442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852410 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89b453d-cce6-4ca8-9133-ab91353f1249" containerName="istio-proxy" Oct 29 22:19:29.852442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.852420 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2081f513-c697-494f-9031-b854d97de08c" containerName="istio-proxy" Oct 29 22:19:29.857787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.857768 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:29.877307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.877242 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e7b56d-serving-cert\"" Oct 29 22:19:29.879552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.879521 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:19:29.965091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.965059 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:29.965266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.965111 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj6k\" (UniqueName: \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:29.965420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.965400 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:29.965462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:29.965434 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.075927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.072199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.075927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.072256 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.075927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.072409 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.075927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.072481 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj6k\" (UniqueName: \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.080975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.078409 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.098342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.098300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.099509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.099063 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.147365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.147283 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj6k\" (UniqueName: \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\") pod \"maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.177547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.177516 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:30.498174 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.498143 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:19:30.498587 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:19:30.498496 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d307021_fd9c_4162_a225_b5e628de2808.slice/crio-84d276edb7d759ba2626cfa2c27ef2b082173a5fc554e71aea6964a3cae0b641 WatchSource:0}: Error finding container 84d276edb7d759ba2626cfa2c27ef2b082173a5fc554e71aea6964a3cae0b641: Status 404 returned error can't find the container with id 84d276edb7d759ba2626cfa2c27ef2b082173a5fc554e71aea6964a3cae0b641 Oct 29 22:19:30.671336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.671300 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mkczb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:19:30.671504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.671367 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:19:30.740447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:30.740408 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" event={"ID":"7d307021-fd9c-4162-a225-b5e628de2808","Type":"ContainerStarted","Data":"84d276edb7d759ba2626cfa2c27ef2b082173a5fc554e71aea6964a3cae0b641"} Oct 29 22:19:31.749689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:31.749643 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" event={"ID":"7d307021-fd9c-4162-a225-b5e628de2808","Type":"ContainerStarted","Data":"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c"} Oct 29 22:19:31.750186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:31.749710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:31.797839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:31.797791 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" podStartSLOduration=2.3472396189999998 podStartE2EDuration="2.797776489s" totalImagesPullingTime="450.53687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:31.794002079 +0000 UTC m=+2230.642478923" watchObservedRunningTime="2026-10-29 22:19:31.797776489 +0000 UTC m=+2230.646253334" Oct 29 22:19:32.157562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.157532 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 22:19:32.206113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.205965 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206088 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206120 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206195 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206221 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206301 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206437 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206493 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.206759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206557 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\") pod \"df087dc1-2f0e-4838-b84c-a7386e69815c\" (UID: \"df087dc1-2f0e-4838-b84c-a7386e69815c\") " Oct 29 22:19:32.207799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:32.207799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206758 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:32.207799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.206963 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:32.207799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.207705 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:32.207799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.207734 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.210702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.210653 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:32.218527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.218456 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:32.218527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.218498 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "kube-api-access-5jt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:32.218814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.218782 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token" pod "df087dc1-2f0e-4838-b84c-a7386e69815c" (UID: "df087dc1-2f0e-4838-b84c-a7386e69815c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309227 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jt47\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-kube-api-access-5jt47\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309262 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309273 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309283 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309291 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309300 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df087dc1-2f0e-4838-b84c-a7386e69815c-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309308 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df087dc1-2f0e-4838-b84c-a7386e69815c-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.309319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.309317 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df087dc1-2f0e-4838-b84c-a7386e69815c-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:32.758287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.758252 4376 generic.go:356] "Generic (PLEG): container finished" podID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerID="e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba" exitCode=0 Oct 29 22:19:32.758785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.758328 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" Oct 29 22:19:32.758785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.758333 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" event={"ID":"df087dc1-2f0e-4838-b84c-a7386e69815c","Type":"ContainerDied","Data":"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba"} Oct 29 22:19:32.758785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.758422 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb" event={"ID":"df087dc1-2f0e-4838-b84c-a7386e69815c","Type":"ContainerDied","Data":"080500c8bf9165868dbec362f20537da6032d539679921d36338fee5cef327e4"} Oct 29 22:19:32.758785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.758453 4376 scope.go:122] "RemoveContainer" containerID="e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba" Oct 29 22:19:32.771086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.771068 4376 scope.go:122] "RemoveContainer" containerID="e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba" Oct 29 22:19:32.771424 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:32.771403 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba\": container with ID starting with e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba not found: ID does not exist" containerID="e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba" Oct 29 22:19:32.771498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.771431 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba"} err="failed to get container status \"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba\": rpc error: code = NotFound desc = could not find container \"e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba\": container with ID starting with e9ec2858402b71f830f390f20327b6a104ae20259d1e73e2099d2b89348f18ba not found: ID does not exist" Oct 29 22:19:32.815895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.815861 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 22:19:32.828597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:32.828571 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mkczb"] Oct 29 22:19:33.274125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:33.274083 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" path="/var/lib/kubelet/pods/df087dc1-2f0e-4838-b84c-a7386e69815c/volumes" Oct 29 22:19:34.260472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:34.260435 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 22:19:34.261228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:34.261184 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" containerID="cri-o://99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8" gracePeriod=30 Oct 29 22:19:37.049131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:37.049080 4376 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:19:37.049727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:37.049165 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:19:37.815127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:37.815090 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:19:39.551348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.551276 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.742949 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743057 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743144 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743214 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743397 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743532 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743586 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.745334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.743627 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\") pod \"26eb8767-d77f-4e57-a298-f7e4e9abe316\" (UID: \"26eb8767-d77f-4e57-a298-f7e4e9abe316\") " Oct 29 22:19:39.746600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.746274 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.835964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.830402 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.837175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.837130 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:39.837306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.837250 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.916215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.870070 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.918044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.917688 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:39.918044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.917726 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:39.918044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.917740 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26eb8767-d77f-4e57-a298-f7e4e9abe316-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:39.918044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.917752 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:39.918044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.917765 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:39.918710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.918670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.920895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.920869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "kube-api-access-5b5ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:39.954923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.950224 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:39.960010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.956239 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo" pod "26eb8767-d77f-4e57-a298-f7e4e9abe316" (UID: "26eb8767-d77f-4e57-a298-f7e4e9abe316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:39.986046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.985220 4376 generic.go:356] "Generic (PLEG): container finished" podID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerID="99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8" exitCode=0 Oct 29 22:19:39.986046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.985325 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerDied","Data":"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8"} Oct 29 22:19:39.986046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.985365 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" event={"ID":"26eb8767-d77f-4e57-a298-f7e4e9abe316","Type":"ContainerDied","Data":"b60035ca811289957d4a8db67844ef5ea688b4035a4e08997c4c76fc97b43a27"} Oct 29 22:19:39.986046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.985400 4376 scope.go:122] "RemoveContainer" containerID="99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8" Oct 29 22:19:39.986046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:39.985548 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll" Oct 29 22:19:40.020726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.020585 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26eb8767-d77f-4e57-a298-f7e4e9abe316-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.020726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.020623 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5ms\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-kube-api-access-5b5ms\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.020726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.020649 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.020726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.020663 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26eb8767-d77f-4e57-a298-f7e4e9abe316-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.039396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.039368 4376 scope.go:122] "RemoveContainer" containerID="8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562" Oct 29 22:19:40.052485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.052464 4376 scope.go:122] "RemoveContainer" containerID="99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8" Oct 29 22:19:40.052788 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:40.052764 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8\": container with ID starting with 99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8 not found: ID does not exist" containerID="99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8" Oct 29 22:19:40.052863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.052800 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8"} err="failed to get container status \"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8\": rpc error: code = NotFound desc = could not find container \"99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8\": container with ID starting with 99149564cd2d5fb43d2bd0c4a2b1cf3b99967cb0292f8ba42e1591c31d07d4f8 not found: ID does not exist" Oct 29 22:19:40.052863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.052819 4376 scope.go:122] "RemoveContainer" containerID="8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562" Oct 29 22:19:40.053165 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:40.053139 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562\": container with ID starting with 8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562 not found: ID does not exist" containerID="8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562" Oct 29 22:19:40.053261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.053171 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562"} err="failed to get container status \"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562\": rpc error: code = NotFound desc = could not find container \"8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562\": container with ID starting with 8fbb913ad48bfadcd4118179091140225754d72cdc3b95630960ccbf8f24d562 not found: ID does not exist" Oct 29 22:19:40.070501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.070471 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 22:19:40.083170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:40.083142 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-c9cll"] Oct 29 22:19:41.274278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:41.274247 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" path="/var/lib/kubelet/pods/26eb8767-d77f-4e57-a298-f7e4e9abe316/volumes" Oct 29 22:19:46.434988 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:19:46.434944 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:19:49.496865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.496832 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:19:49.497619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497603 4376 cpu_manager.go:414] "containerMap: removing container" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:49.497663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497620 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:49.497663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497634 4376 cpu_manager.go:414] "containerMap: removing container" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" Oct 29 22:19:49.497663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497639 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" Oct 29 22:19:49.497663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497652 4376 cpu_manager.go:414] "containerMap: removing container" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:49.497663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497657 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:49.497830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497755 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="df087dc1-2f0e-4838-b84c-a7386e69815c" containerName="istio-proxy" Oct 29 22:19:49.497830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.497766 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:49.501794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.501768 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.541927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.539395 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-e2e--e23b4f7e-dockercfg-hn4pl\"" Oct 29 22:19:49.692012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.673979 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"openshift-service-ca.crt\"" Oct 29 22:19:49.692012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.674141 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"routing-overlay\"" Oct 29 22:19:49.692012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.674255 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d8760\"" Oct 29 22:19:49.692012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.674395 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"kube-root-ca.crt\"" Oct 29 22:19:49.692012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.674497 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"agc-cred-91745e96a35f\"" Oct 29 22:19:49.780976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.780483 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.780976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.780781 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.780976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.780846 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.781269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.780988 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bk4\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.788303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.786089 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:19:49.977069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.975319 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.977361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.977337 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.978024 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.977999 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.978264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.978245 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bk4\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.979961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.979899 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:49.996498 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:49.980922 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:50.034002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:50.019732 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bk4\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:50.034002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:50.031116 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:50.287392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:50.287305 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:50.605107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:50.605077 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:19:50.605530 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:19:50.605503 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf28ada4_0968_4efd_8049_9b3fe37b31c8.slice/crio-9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9 WatchSource:0}: Error finding container 9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9: Status 404 returned error can't find the container with id 9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9 Oct 29 22:19:51.212578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:51.212540 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" event={"ID":"df28ada4-0968-4efd-8049-9b3fe37b31c8","Type":"ContainerStarted","Data":"31167bcdd9f34f450c30d0dc6152fed60c2007d212ff48d06078fdca37b6af87"} Oct 29 22:19:51.212769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:51.212601 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" event={"ID":"df28ada4-0968-4efd-8049-9b3fe37b31c8","Type":"ContainerStarted","Data":"9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9"} Oct 29 22:19:51.212769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:51.212627 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:19:51.258430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:51.258366 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" podStartSLOduration=2.258354019 podStartE2EDuration="2.258354019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:51.255817296 +0000 UTC m=+2250.104294140" watchObservedRunningTime="2026-10-29 22:19:51.258354019 +0000 UTC m=+2250.106830860" Oct 29 22:19:59.328662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.328624 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:19:59.329701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.329678 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="26eb8767-d77f-4e57-a298-f7e4e9abe316" containerName="istio-proxy" Oct 29 22:19:59.334689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.334667 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.354006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.353739 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-07e166-serving-cert\"" Oct 29 22:19:59.357542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.357469 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:19:59.452566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.452527 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.452736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.452688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.452783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.452774 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.452851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.452839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhbv\" (UniqueName: \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.561923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.556199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.561923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.556313 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.561923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.556396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhbv\" (UniqueName: \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.561923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.556630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.565926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.563667 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.595972 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.591375 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.613239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.613203 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.613239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.613236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhbv\" (UniqueName: \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\") pod \"maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.655043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.655012 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:19:59.964889 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:19:59.964859 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814c9c42_6fba_4eea_887a_8d4acc7c26a9.slice/crio-925ae324a60e1bc5634e730c16901cc3d381208dae35686b14994a702a5a5a2b WatchSource:0}: Error finding container 925ae324a60e1bc5634e730c16901cc3d381208dae35686b14994a702a5a5a2b: Status 404 returned error can't find the container with id 925ae324a60e1bc5634e730c16901cc3d381208dae35686b14994a702a5a5a2b Oct 29 22:19:59.965017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:19:59.964893 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:20:00.277877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:00.277791 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" event={"ID":"814c9c42-6fba-4eea-887a-8d4acc7c26a9","Type":"ContainerStarted","Data":"925ae324a60e1bc5634e730c16901cc3d381208dae35686b14994a702a5a5a2b"} Oct 29 22:20:01.289146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:01.289111 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" event={"ID":"814c9c42-6fba-4eea-887a-8d4acc7c26a9","Type":"ContainerStarted","Data":"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb"} Oct 29 22:20:01.289577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:01.289318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:20:01.336190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:01.336138 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" podStartSLOduration=1.96607989 podStartE2EDuration="2.336124245s" totalImagesPullingTime="370.044355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:01.333152412 +0000 UTC m=+2260.181629256" watchObservedRunningTime="2026-10-29 22:20:01.336124245 +0000 UTC m=+2260.184601089" Oct 29 22:20:02.223560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:02.223528 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:20:07.305387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:07.305358 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:20:12.965862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.965823 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:20:12.966340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.966049 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" containerID="cri-o://5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a" gracePeriod=30 Oct 29 22:20:12.977982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.977948 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 22:20:12.978170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.978149 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" containerID="cri-o://f9330b7e0a3f2e6ef4b34e4fa2edb9907ed73268205438505a6bbe74f3a55d4e" gracePeriod=30 Oct 29 22:20:12.982504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.982477 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 22:20:12.982687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.982663 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" containerID="cri-o://a2d614f5007a4b94982331fa2fb43010e0e02ff8ca04228ba70658ff89d93d4e" gracePeriod=30 Oct 29 22:20:12.991113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.991081 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:20:12.991282 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:12.991262 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" containerID="cri-o://ca8a8a534c3816d4e2a506725e6355a880f52a14daa5781a7d26ae390e1550a2" gracePeriod=30 Oct 29 22:20:16.436813 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:20:16.436769 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:20:18.316850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.316821 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:20:18.372761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.372060 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.386134 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.386194 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.386265 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.386326 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.386406 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.388878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.387502 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.394168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.394134 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.455931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.451144 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.455931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.451355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.455931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.451430 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.455931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.451501 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\") pod \"d39390aa-e691-498c-971b-2e928c2a2f5e\" (UID: \"d39390aa-e691-498c-971b-2e928c2a2f5e\") " Oct 29 22:20:18.467688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.461614 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:18.467688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.462306 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.472800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.471546 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.473958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.473747 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "kube-api-access-g5ljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.475947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.474123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.478164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.477192 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:18.658892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.658864 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659124 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659224 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659312 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ljj\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-kube-api-access-g5ljj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659410 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659399 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659479 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.659573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.659563 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.685274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.685238 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 22:20:18.689031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.688886 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy" pod "d39390aa-e691-498c-971b-2e928c2a2f5e" (UID: "d39390aa-e691-498c-971b-2e928c2a2f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.690653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.690628 4376 generic.go:356] "Generic (PLEG): container finished" podID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerID="ca8a8a534c3816d4e2a506725e6355a880f52a14daa5781a7d26ae390e1550a2" exitCode=0 Oct 29 22:20:18.690952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.690931 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" event={"ID":"3e84ca7b-3a73-4053-9b71-702d4215a8b9","Type":"ContainerDied","Data":"ca8a8a534c3816d4e2a506725e6355a880f52a14daa5781a7d26ae390e1550a2"} Oct 29 22:20:18.692685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.692667 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:20:18.693055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.693040 4376 generic.go:356] "Generic (PLEG): container finished" podID="a8676ade-b183-4ada-9a0a-94122c52b341" containerID="f9330b7e0a3f2e6ef4b34e4fa2edb9907ed73268205438505a6bbe74f3a55d4e" exitCode=0 Oct 29 22:20:18.693249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.693104 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" Oct 29 22:20:18.693423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.693126 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh" event={"ID":"a8676ade-b183-4ada-9a0a-94122c52b341","Type":"ContainerDied","Data":"f9330b7e0a3f2e6ef4b34e4fa2edb9907ed73268205438505a6bbe74f3a55d4e"} Oct 29 22:20:18.693515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.693506 4376 scope.go:122] "RemoveContainer" containerID="f9330b7e0a3f2e6ef4b34e4fa2edb9907ed73268205438505a6bbe74f3a55d4e" Oct 29 22:20:18.695625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.695607 4376 generic.go:356] "Generic (PLEG): container finished" podID="64cdab92-ac02-434a-97fd-d6df904b2365" containerID="a2d614f5007a4b94982331fa2fb43010e0e02ff8ca04228ba70658ff89d93d4e" exitCode=0 Oct 29 22:20:18.695707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.695683 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerDied","Data":"a2d614f5007a4b94982331fa2fb43010e0e02ff8ca04228ba70658ff89d93d4e"} Oct 29 22:20:18.696670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.696650 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 22:20:18.697264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.697245 4376 generic.go:356] "Generic (PLEG): container finished" podID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerID="5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a" exitCode=0 Oct 29 22:20:18.697337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.697291 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" Oct 29 22:20:18.697375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.697328 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" event={"ID":"d39390aa-e691-498c-971b-2e928c2a2f5e","Type":"ContainerDied","Data":"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a"} Oct 29 22:20:18.697413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.697378 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5" event={"ID":"d39390aa-e691-498c-971b-2e928c2a2f5e","Type":"ContainerDied","Data":"0258e0dc07607f4871b8bfde04bf34c7a3edc7055de33a0b7eb373c40b52f62c"} Oct 29 22:20:18.708351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.708331 4376 scope.go:122] "RemoveContainer" containerID="d1ecc8e5a9251cdc26519b74a76428c5d0bbe202185efe2c5830e282b782d269" Oct 29 22:20:18.721274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.721248 4376 scope.go:122] "RemoveContainer" containerID="5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a" Oct 29 22:20:18.736844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.736810 4376 scope.go:122] "RemoveContainer" containerID="5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a" Oct 29 22:20:18.741948 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:20:18.737219 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a\": container with ID starting with 5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a not found: ID does not exist" containerID="5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a" Oct 29 22:20:18.741948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.737259 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a"} err="failed to get container status \"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a\": rpc error: code = NotFound desc = could not find container \"5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a\": container with ID starting with 5ed6e6038e46eccf5ca62664919077ddbfac9358f859d277646400ec9ac9ab4a not found: ID does not exist" Oct 29 22:20:18.760680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760637 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.760843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760685 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.760843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760717 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.760843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760766 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.760843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760813 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760852 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760882 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.761132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.760999 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.761132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761064 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.761341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761128 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761217 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761251 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.761341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761326 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761382 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761452 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.761604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761489 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761530 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.761604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761591 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.761855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761658 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761743 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.761855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761782 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.761855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761836 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.762110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761883 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.762110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761942 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\") pod \"64cdab92-ac02-434a-97fd-d6df904b2365\" (UID: \"64cdab92-ac02-434a-97fd-d6df904b2365\") " Oct 29 22:20:18.762110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.761970 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.762110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.762043 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\") pod \"a8676ade-b183-4ada-9a0a-94122c52b341\" (UID: \"a8676ade-b183-4ada-9a0a-94122c52b341\") " Oct 29 22:20:18.762680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.762651 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.763635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.763609 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\") pod \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\" (UID: \"3e84ca7b-3a73-4053-9b71-702d4215a8b9\") " Oct 29 22:20:18.764651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764623 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764657 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39390aa-e691-498c-971b-2e928c2a2f5e-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764672 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39390aa-e691-498c-971b-2e928c2a2f5e-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764684 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764697 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.763615 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764013 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764147 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764295 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764331 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764544 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.764774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764640 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.765386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.764838 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.765386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.765187 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:18.765787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.765755 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.766029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.766006 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.767131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.766847 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.767339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.767309 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.769283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.769246 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:18.769395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.769317 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "kube-api-access-s5k2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.770191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.769996 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.770191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770067 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.770191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770097 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.770191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770143 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "kube-api-access-nw7s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.770191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770161 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:18.770511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770211 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy" pod "3e84ca7b-3a73-4053-9b71-702d4215a8b9" (UID: "3e84ca7b-3a73-4053-9b71-702d4215a8b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.770988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.770964 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq" pod "64cdab92-ac02-434a-97fd-d6df904b2365" (UID: "64cdab92-ac02-434a-97fd-d6df904b2365"). InnerVolumeSpecName "kube-api-access-6btjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.771102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.771010 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:18.771961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.771896 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy" pod "a8676ade-b183-4ada-9a0a-94122c52b341" (UID: "a8676ade-b183-4ada-9a0a-94122c52b341"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.829635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.829114 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:20:18.836569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.836536 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-j4ql5"] Oct 29 22:20:18.866095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866061 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866090 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866102 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866111 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866119 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866130 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8676ade-b183-4ada-9a0a-94122c52b341-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866141 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866153 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866164 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866172 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866181 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6btjq\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-kube-api-access-6btjq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866189 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866197 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5k2l\" (UniqueName: \"kubernetes.io/projected/a8676ade-b183-4ada-9a0a-94122c52b341-kube-api-access-s5k2l\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866205 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866213 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866221 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cdab92-ac02-434a-97fd-d6df904b2365-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866232 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cdab92-ac02-434a-97fd-d6df904b2365-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866239 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e84ca7b-3a73-4053-9b71-702d4215a8b9-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866246 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866254 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7s8\" (UniqueName: \"kubernetes.io/projected/3e84ca7b-3a73-4053-9b71-702d4215a8b9-kube-api-access-nw7s8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866262 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cdab92-ac02-434a-97fd-d6df904b2365-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866271 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cdab92-ac02-434a-97fd-d6df904b2365-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866278 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8676ade-b183-4ada-9a0a-94122c52b341-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.866346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:18.866286 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8676ade-b183-4ada-9a0a-94122c52b341-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:19.077824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.077784 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 22:20:19.084529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.084449 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-h8lvh"] Oct 29 22:20:19.274173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.274136 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" path="/var/lib/kubelet/pods/a8676ade-b183-4ada-9a0a-94122c52b341/volumes" Oct 29 22:20:19.274766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.274744 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" path="/var/lib/kubelet/pods/d39390aa-e691-498c-971b-2e928c2a2f5e/volumes" Oct 29 22:20:19.706855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.706827 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" Oct 29 22:20:19.707353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.706862 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d" event={"ID":"64cdab92-ac02-434a-97fd-d6df904b2365","Type":"ContainerDied","Data":"e4657668d7d6b5f21b6bf77fce9c7aad52b6d87169989e635905ac0d66c1bad3"} Oct 29 22:20:19.707353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.706966 4376 scope.go:122] "RemoveContainer" containerID="a2d614f5007a4b94982331fa2fb43010e0e02ff8ca04228ba70658ff89d93d4e" Oct 29 22:20:19.710426 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.710401 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" Oct 29 22:20:19.710426 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.710410 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8" event={"ID":"3e84ca7b-3a73-4053-9b71-702d4215a8b9","Type":"ContainerDied","Data":"2dd4942e949613d3a64a431a40d63571d5ee0702f8bd8547d17c42061f8482c2"} Oct 29 22:20:19.719764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.719741 4376 scope.go:122] "RemoveContainer" containerID="ca8a8a534c3816d4e2a506725e6355a880f52a14daa5781a7d26ae390e1550a2" Oct 29 22:20:19.794601 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.794566 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 22:20:19.800151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.800123 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-txs6d"] Oct 29 22:20:19.839217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.839181 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:20:19.842204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:19.842182 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-vrwg8"] Oct 29 22:20:21.274702 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:21.274669 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" path="/var/lib/kubelet/pods/3e84ca7b-3a73-4053-9b71-702d4215a8b9/volumes" Oct 29 22:20:21.275354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:21.275337 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" path="/var/lib/kubelet/pods/64cdab92-ac02-434a-97fd-d6df904b2365/volumes" Oct 29 22:20:25.700817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.700772 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:20:25.701751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701724 4376 cpu_manager.go:414] "containerMap: removing container" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:25.701751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701751 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701810 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701819 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701839 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701847 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701867 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" Oct 29 22:20:25.701960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.701875 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" Oct 29 22:20:25.702288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.702087 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39390aa-e691-498c-971b-2e928c2a2f5e" containerName="istio-proxy" Oct 29 22:20:25.702288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.702108 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8676ade-b183-4ada-9a0a-94122c52b341" containerName="istio-proxy" Oct 29 22:20:25.702288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.702121 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:25.702288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.702135 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:25.702288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.702152 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e84ca7b-3a73-4053-9b71-702d4215a8b9" containerName="istio-proxy" Oct 29 22:20:25.708422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.708400 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.725790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.725767 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-5a079a386429\"" Oct 29 22:20:25.737339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.737318 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e7b56d\"" Oct 29 22:20:25.737440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.737369 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"routing-overlay\"" Oct 29 22:20:25.737440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.737376 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"kube-root-ca.crt\"" Oct 29 22:20:25.737605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.737584 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-e2e--885c5d1d-dockercfg-4jm2b\"" Oct 29 22:20:25.739171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.739154 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"openshift-service-ca.crt\"" Oct 29 22:20:25.752635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.752612 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:20:25.765584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.765554 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.765767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.765748 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.765939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.765914 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9g4\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.766018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.766003 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.867585 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.867542 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.867791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.867696 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.867791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.867787 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9g4\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.867959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.867854 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.868878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.868851 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.869495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.869472 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.914751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.901761 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:25.914751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:25.911661 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9g4\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:26.023695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.023349 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:26.431467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.431436 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:20:26.432059 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:26.432035 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11381e51_f9a6_457f_989e_6965ae2609da.slice/crio-6bb3eb37099619f5441b4194e7802c21ca471ed39f9d623b8f56dc9801748481 WatchSource:0}: Error finding container 6bb3eb37099619f5441b4194e7802c21ca471ed39f9d623b8f56dc9801748481: Status 404 returned error can't find the container with id 6bb3eb37099619f5441b4194e7802c21ca471ed39f9d623b8f56dc9801748481 Oct 29 22:20:26.767004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.766960 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" event={"ID":"11381e51-f9a6-457f-989e-6965ae2609da","Type":"ContainerStarted","Data":"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a"} Oct 29 22:20:26.767421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.767030 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" event={"ID":"11381e51-f9a6-457f-989e-6965ae2609da","Type":"ContainerStarted","Data":"6bb3eb37099619f5441b4194e7802c21ca471ed39f9d623b8f56dc9801748481"} Oct 29 22:20:26.767421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.767210 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:26.827429 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:26.827380 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" podStartSLOduration=1.827367739 podStartE2EDuration="1.827367739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:26.826701442 +0000 UTC m=+2285.675178299" watchObservedRunningTime="2026-10-29 22:20:26.827367739 +0000 UTC m=+2285.675844584" Oct 29 22:20:33.496925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:33.491044 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerName="main" containerID="cri-o://82347169bb4e9248056be9ada301b8be1e99f32e6f50f4c01d4b37d0646b24ee" gracePeriod=45 Oct 29 22:20:33.741926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:33.741365 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": dial tcp 10.134.0.161:8000: connect: connection refused" Oct 29 22:20:34.267966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.249640 4376 generic.go:356] "Generic (PLEG): container finished" podID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerID="82347169bb4e9248056be9ada301b8be1e99f32e6f50f4c01d4b37d0646b24ee" exitCode=0 Oct 29 22:20:34.267966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.249794 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" event={"ID":"580bb07a-d7d7-4d59-81d6-620c68eeb26f","Type":"ContainerDied","Data":"82347169bb4e9248056be9ada301b8be1e99f32e6f50f4c01d4b37d0646b24ee"} Oct 29 22:20:34.299925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.276698 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:20:34.299925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.277940 4376 cpu_manager.go:414] "containerMap: removing container" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:34.299925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.277963 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cdab92-ac02-434a-97fd-d6df904b2365" containerName="istio-proxy" Oct 29 22:20:34.402349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.395543 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:20:34.402349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.396052 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.456942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.454224 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:20:34.469636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.469548 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-7fe78e-openshift-default-dockercfg-lfgv8\"" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.536931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.537186 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.537619 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.537945 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.538020 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.538048 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhm4t\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.538121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.538278 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.539979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.538333 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.683773 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684086 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684162 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684268 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684472 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\" (UniqueName: \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684559 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\" (UniqueName: \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\") pod \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\" (UID: \"580bb07a-d7d7-4d59-81d6-620c68eeb26f\") " Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.684950 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.693927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.685023 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.707928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.707857 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.708114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708026 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.708178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.708241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708221 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.711923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708296 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.711923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708334 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhm4t\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.711923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.708454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.724926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.714141 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.724926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.714499 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.737286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.729502 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.737286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.729885 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.737286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.730038 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:34.741843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.741811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:34.742269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.742247 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:34.743659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.743613 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.753586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.747104 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "kube-api-access-jb6p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:34.753586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.748872 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.754270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.754247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.755440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.755063 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:20:34.755440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.755077 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm" pod "580bb07a-d7d7-4d59-81d6-620c68eeb26f" (UID: "580bb07a-d7d7-4d59-81d6-620c68eeb26f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:34.758141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.758097 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.780232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.780160 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhm4t\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\") pod \"e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:34.810270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810238 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:34.810270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810268 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:34.810270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810277 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:34.810520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810285 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580bb07a-d7d7-4d59-81d6-620c68eeb26f-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:34.810520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810294 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb6p5\" (UniqueName: \"kubernetes.io/projected/580bb07a-d7d7-4d59-81d6-620c68eeb26f-kube-api-access-jb6p5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:34.810520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:34.810304 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580bb07a-d7d7-4d59-81d6-620c68eeb26f-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:35.045659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.045617 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:35.258736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.258708 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" Oct 29 22:20:35.258938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.258739 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx" event={"ID":"580bb07a-d7d7-4d59-81d6-620c68eeb26f","Type":"ContainerDied","Data":"2764af5c1166a3e756d438a9501548282c33b9561e7882eee8726c00a37e335e"} Oct 29 22:20:35.258938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.258789 4376 scope.go:122] "RemoveContainer" containerID="82347169bb4e9248056be9ada301b8be1e99f32e6f50f4c01d4b37d0646b24ee" Oct 29 22:20:35.339442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.339352 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:20:35.349992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.349957 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05687d/auth-test-model-05687d-kserve-5c4d85cc65-fc6zx"] Oct 29 22:20:35.391133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.391107 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:20:35.391616 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:35.391588 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27fa9f4_2bc6_406a_95c3_959ec1d2361d.slice/crio-0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717 WatchSource:0}: Error finding container 0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717: Status 404 returned error can't find the container with id 0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717 Oct 29 22:20:35.394522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.394491 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:35.394605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.394557 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:35.394605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:35.394589 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:36.266696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:36.266660 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerStarted","Data":"2d4cde3911e106537ab83dfbd84748ade7c4e51f1898a33300683f24ca837bbc"} Oct 29 22:20:36.267157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:36.266720 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerStarted","Data":"0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717"} Oct 29 22:20:36.342748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:36.342687 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podStartSLOduration=2.342674535 podStartE2EDuration="2.342674535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:36.328009972 +0000 UTC m=+2295.176486830" watchObservedRunningTime="2026-10-29 22:20:36.342674535 +0000 UTC m=+2295.191151380" Oct 29 22:20:37.046424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.046389 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:37.050653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.050624 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:20:37.050841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.050819 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" containerID="cri-o://f59613b123104fedd40a7ae773a5c28bf3c50e6bf290bf0cb3746158ea7c4720" gracePeriod=30 Oct 29 22:20:37.052729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.052707 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:37.274404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.274361 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" path="/var/lib/kubelet/pods/580bb07a-d7d7-4d59-81d6-620c68eeb26f/volumes" Oct 29 22:20:37.275114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.275079 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:37.276132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.276114 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:20:37.776345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:37.776318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:20:40.660979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.660936 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:20:40.661766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.661749 4376 cpu_manager.go:414] "containerMap: removing container" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerName="main" Oct 29 22:20:40.661836 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.661769 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerName="main" Oct 29 22:20:40.661896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.661887 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="580bb07a-d7d7-4d59-81d6-620c68eeb26f" containerName="main" Oct 29 22:20:40.666837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.666816 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.684287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.684251 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:20:40.691040 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.691007 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-7fe78e\"" Oct 29 22:20:40.693022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.692996 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-7fe78e-dockercfg-hb7cm\"" Oct 29 22:20:40.719287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.719244 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5f4\" (UniqueName: \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.719497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.719363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.784125 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.784042 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:20:40.828387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.828314 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5f4\" (UniqueName: \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.828695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.828670 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.830257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.830219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.894029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.893986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5f4\" (UniqueName: \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\") pod \"payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:40.895833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.895804 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:20:41.003107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.901032 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.004321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.954865 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.005688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.005644 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjdk\" (UniqueName: \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.012712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:40.984532 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:41.142935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.135636 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjdk\" (UniqueName: \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.142935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.135737 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.142935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.139787 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.183207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.181562 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjdk\" (UniqueName: \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\") pod \"payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.351760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.351675 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:41.431379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.431347 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:20:41.434032 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:41.434000 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b17f744_3e78_4206_9817_ff7482e3ab49.slice/crio-b6cfae7f23547e7aeca4a10d10e11d4c7cca5219e793fd2acd21a96ef2084f7b WatchSource:0}: Error finding container b6cfae7f23547e7aeca4a10d10e11d4c7cca5219e793fd2acd21a96ef2084f7b: Status 404 returned error can't find the container with id b6cfae7f23547e7aeca4a10d10e11d4c7cca5219e793fd2acd21a96ef2084f7b Oct 29 22:20:41.663604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:41.663571 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:20:41.664103 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:41.664080 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbea50e9_6d66_4ae1_ba71_dd29b98795cc.slice/crio-146bca5869ccc4d0e5703459157d9adcc0a422b3d23c6b4716d022f362166454 WatchSource:0}: Error finding container 146bca5869ccc4d0e5703459157d9adcc0a422b3d23c6b4716d022f362166454: Status 404 returned error can't find the container with id 146bca5869ccc4d0e5703459157d9adcc0a422b3d23c6b4716d022f362166454 Oct 29 22:20:42.312673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.312634 4376 generic.go:356] "Generic (PLEG): container finished" podID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerID="f59613b123104fedd40a7ae773a5c28bf3c50e6bf290bf0cb3746158ea7c4720" exitCode=0 Oct 29 22:20:42.312842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.312697 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" event={"ID":"7aa24378-8958-4bbb-818f-bc36e45aca27","Type":"ContainerDied","Data":"f59613b123104fedd40a7ae773a5c28bf3c50e6bf290bf0cb3746158ea7c4720"} Oct 29 22:20:42.312842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.312749 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" event={"ID":"7aa24378-8958-4bbb-818f-bc36e45aca27","Type":"ContainerDied","Data":"01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb"} Oct 29 22:20:42.312842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.312770 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f4233ee643009d809cd8b923d1325f1c096264837759e79654abfd8b151ddb" Oct 29 22:20:42.314420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.314395 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" event={"ID":"7b17f744-3e78-4206-9817-ff7482e3ab49","Type":"ContainerStarted","Data":"444116aa9405a7628a2eac9a9ed681f495a942c7a0f79dceb8ea3eb15a21e1a5"} Oct 29 22:20:42.314556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.314442 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" event={"ID":"7b17f744-3e78-4206-9817-ff7482e3ab49","Type":"ContainerStarted","Data":"b6cfae7f23547e7aeca4a10d10e11d4c7cca5219e793fd2acd21a96ef2084f7b"} Oct 29 22:20:42.314556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.314472 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:42.316119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.316100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" event={"ID":"dbea50e9-6d66-4ae1-ba71-dd29b98795cc","Type":"ContainerStarted","Data":"79c62799ee7997b36a12375e74918db3b3e4b3c845708e5b30392ec99d58200a"} Oct 29 22:20:42.316167 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.316138 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" event={"ID":"dbea50e9-6d66-4ae1-ba71-dd29b98795cc","Type":"ContainerStarted","Data":"146bca5869ccc4d0e5703459157d9adcc0a422b3d23c6b4716d022f362166454"} Oct 29 22:20:42.316260 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.316244 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:42.341346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.341283 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:20:42.356880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.356826 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" podStartSLOduration=2.356812281 podStartE2EDuration="2.356812281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:42.356347525 +0000 UTC m=+2301.204824371" watchObservedRunningTime="2026-10-29 22:20:42.356812281 +0000 UTC m=+2301.205289127" Oct 29 22:20:42.399344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.399278 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" podStartSLOduration=2.399261706 podStartE2EDuration="2.399261706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:42.397547064 +0000 UTC m=+2301.246023909" watchObservedRunningTime="2026-10-29 22:20:42.399261706 +0000 UTC m=+2301.247738551" Oct 29 22:20:42.483443 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.483404 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.483637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.483482 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.483953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.483926 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:42.484078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484059 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484245 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484519 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:42.484623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484561 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484600 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484666 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.484824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.484725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\") pod \"7aa24378-8958-4bbb-818f-bc36e45aca27\" (UID: \"7aa24378-8958-4bbb-818f-bc36e45aca27\") " Oct 29 22:20:42.485537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.485513 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.485625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.485541 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa24378-8958-4bbb-818f-bc36e45aca27-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.485760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.485740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:42.486272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.486245 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:42.486517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.486493 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:42.486589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.486524 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:42.486741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.486722 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "kube-api-access-rcfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:42.488204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.488164 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:42.488512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.488484 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo" pod "7aa24378-8958-4bbb-818f-bc36e45aca27" (UID: "7aa24378-8958-4bbb-818f-bc36e45aca27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613683 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613719 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613734 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613746 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613762 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613775 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfqx\" (UniqueName: \"kubernetes.io/projected/7aa24378-8958-4bbb-818f-bc36e45aca27-kube-api-access-rcfqx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:42.613873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:42.613787 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa24378-8958-4bbb-818f-bc36e45aca27-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:20:43.323860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:43.323824 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6" Oct 29 22:20:43.385233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:43.385198 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:20:43.398099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:43.398074 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-rgsm6"] Oct 29 22:20:45.273138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:45.273105 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" path="/var/lib/kubelet/pods/7aa24378-8958-4bbb-818f-bc36e45aca27/volumes" Oct 29 22:20:46.483057 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:20:46.483016 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:20:50.372871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.372834 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:20:50.374298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.374271 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" Oct 29 22:20:50.374298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.374296 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" Oct 29 22:20:50.374539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.374523 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa24378-8958-4bbb-818f-bc36e45aca27" containerName="istio-proxy" Oct 29 22:20:50.385948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.379997 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.407572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.407546 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:20:50.410332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.410302 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b9ed94-openshift-default-dockercfg-7pdjl\"" Oct 29 22:20:50.590719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.590677 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.591046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.591019 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.591310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.591293 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.591450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.591436 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.594928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.591555 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7krs\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.623954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.623841 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.624274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.624251 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.624468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.624452 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.624845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.624824 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.730215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.730158 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.730517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.730490 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7krs\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.730640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.730625 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.730756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.730742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.730896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.730882 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.731247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.731225 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.731612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.731545 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.731891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.731867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.774181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.745426 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.774424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.747029 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.774554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.747312 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.774667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.748170 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.774773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.756441 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.775144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.775119 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.781631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.775554 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.789549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.789489 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.805531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.805498 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7krs\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:50.806405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:50.806360 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:51.002977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.002940 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:51.363724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.363697 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:20:51.366612 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:51.366584 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c5c73a_3d11_4406_b921_d053f710c6da.slice/crio-daa79dd2a92bb9f10c2e1420905eabc0997f4045e84f6dd737db371dcaad5d76 WatchSource:0}: Error finding container daa79dd2a92bb9f10c2e1420905eabc0997f4045e84f6dd737db371dcaad5d76: Status 404 returned error can't find the container with id daa79dd2a92bb9f10c2e1420905eabc0997f4045e84f6dd737db371dcaad5d76 Oct 29 22:20:51.370025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.369978 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:51.370133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.370069 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:51.370133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.370112 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:51.579860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.579828 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerStarted","Data":"18941031fcdc24f700a1d0abaa522fe5e7435110adeeb982aa4125ab34086b69"} Oct 29 22:20:51.581104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.580071 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerStarted","Data":"daa79dd2a92bb9f10c2e1420905eabc0997f4045e84f6dd737db371dcaad5d76"} Oct 29 22:20:51.632668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:51.632548 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podStartSLOduration=1.632531644 podStartE2EDuration="1.632531644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:51.631610769 +0000 UTC m=+2310.480087612" watchObservedRunningTime="2026-10-29 22:20:51.632531644 +0000 UTC m=+2310.481008491" Oct 29 22:20:52.003423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:52.003392 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:52.005078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:52.005052 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:20:52.005197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:52.005100 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:20:53.009058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:53.009026 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:53.327436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:53.327359 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:20:53.327436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:53.327426 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:20:53.595346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:53.595262 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:53.596276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:53.596256 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:20:58.776414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.776375 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:20:58.796512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.796486 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.807361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.807329 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:20:58.812960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.812934 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-76c1b7-openshift-default-dockercfg-rvvsj\"" Oct 29 22:20:58.927494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927455 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927609 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927655 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927866 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927920 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.927989 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69nq\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.928152 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.929062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.928180 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:58.956830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.956804 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": read tcp 10.134.0.2:56290->10.134.0.178:15021: read: connection reset by peer" start-of-body= Oct 29 22:20:58.957052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:58.957024 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": read tcp 10.134.0.2:56290->10.134.0.178:15021: read: connection reset by peer" Oct 29 22:20:59.029487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029501 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029647 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029679 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029738 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029769 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69nq\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029861 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.029970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.030082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.029980 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.030082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.030010 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.030082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.030011 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.030295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.030277 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.030737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.030713 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.032099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.032072 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.032433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.032415 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.054557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.054531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.054660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.054626 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69nq\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.116814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.116731 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:20:59.455161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.451020 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:20:59.471284 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:20:59.471248 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f93c982_c57f_4d1d_94b8_31c9a7ee7b93.slice/crio-e6b3b0d8719ba07ee8861a481408e6855316aceeab33144df13a343a349bd090 WatchSource:0}: Error finding container e6b3b0d8719ba07ee8861a481408e6855316aceeab33144df13a343a349bd090: Status 404 returned error can't find the container with id e6b3b0d8719ba07ee8861a481408e6855316aceeab33144df13a343a349bd090 Oct 29 22:20:59.477264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.477227 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.477365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.477305 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.477365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.477341 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.721390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.721284 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" event={"ID":"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93","Type":"ContainerStarted","Data":"720b533e7b115f80198d89f98349784c543552b46457b7919b1edc5d43e66913"} Oct 29 22:20:59.721390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.721358 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" event={"ID":"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93","Type":"ContainerStarted","Data":"e6b3b0d8719ba07ee8861a481408e6855316aceeab33144df13a343a349bd090"} Oct 29 22:20:59.723354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.723321 4376 generic.go:356] "Generic (PLEG): container finished" podID="c9d2bc07-2df6-4758-b256-995e638df935" containerID="c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1" exitCode=0 Oct 29 22:20:59.723511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.723391 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerDied","Data":"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1"} Oct 29 22:20:59.723869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.723850 4376 scope.go:122] "RemoveContainer" containerID="c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1" Oct 29 22:20:59.725755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.725725 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.725838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.725797 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.725838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.725831 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:20:59.768531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:20:59.768466 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podStartSLOduration=1.768448324 podStartE2EDuration="1.768448324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:59.767482918 +0000 UTC m=+2318.615959763" watchObservedRunningTime="2026-10-29 22:20:59.768448324 +0000 UTC m=+2318.616925170" Oct 29 22:21:00.117295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:00.117201 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:21:00.119010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:00.118956 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:00.119157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:00.119010 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:00.732761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:00.732727 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerStarted","Data":"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257"} Oct 29 22:21:01.120979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:01.117219 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:01.120979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:01.117280 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:01.445983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:01.445291 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:21:01.445983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:01.445425 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:21:01.445983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:01.445469 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:21:02.117827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:02.117785 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:02.118072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:02.117839 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:02.454007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:02.445283 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:21:02.454007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:02.445352 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:21:03.118084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:03.118052 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:03.118276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:03.118113 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:03.444983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:03.442874 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:21:03.444983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:03.442968 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:21:04.117167 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:04.117130 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:04.117607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:04.117194 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:04.433767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:04.433667 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:21:04.433767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:04.433733 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:21:05.118049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:05.118020 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:05.118565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:05.118064 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:05.460927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:05.460825 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:21:05.461113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:05.461009 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:21:06.117992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:06.117948 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 29 22:21:06.118189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:06.118020 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 29 22:21:06.438128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:06.438053 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:21:06.438418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:06.438393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:21:06.439078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:06.439061 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:21:07.123228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:07.123195 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:21:07.123685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:07.123664 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:21:07.124658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:07.124635 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:21:16.434424 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:21:16.434385 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:21:24.704863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:21:24.704826 4376 scope.go:122] "RemoveContainer" containerID="f59613b123104fedd40a7ae773a5c28bf3c50e6bf290bf0cb3746158ea7c4720" Oct 29 22:21:46.434741 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:21:46.434697 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:22:02.598157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:02.598113 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:22:08.772988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:08.772519 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:22:08.772988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:08.772803 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" containerID="cri-o://b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea" gracePeriod=30 Oct 29 22:22:14.316154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.316120 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:22:14.346012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.345966 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346095 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346165 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346250 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.346384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346321 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346380 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.346481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346455 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346504 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346568 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346613 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.346679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.346666 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\") pod \"37000434-a758-4adc-8948-b541123008da\" (UID: \"37000434-a758-4adc-8948-b541123008da\") " Oct 29 22:22:14.347782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.347048 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.347782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.347483 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:14.349202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.348267 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.350661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350539 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:14.350661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350590 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37000434-a758-4adc-8948-b541123008da-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.350661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350611 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.350661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350652 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.350969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350666 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.350969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.350680 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.351074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.351002 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:22:14.373199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.373159 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "kube-api-access-shqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:14.373368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.373237 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy" pod "37000434-a758-4adc-8948-b541123008da" (UID: "37000434-a758-4adc-8948-b541123008da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.454947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.454724 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37000434-a758-4adc-8948-b541123008da-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.454947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.454800 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.454947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.454834 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqt2\" (UniqueName: \"kubernetes.io/projected/37000434-a758-4adc-8948-b541123008da-kube-api-access-shqt2\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.454947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.454849 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37000434-a758-4adc-8948-b541123008da-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.591929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.587345 4376 generic.go:356] "Generic (PLEG): container finished" podID="37000434-a758-4adc-8948-b541123008da" containerID="b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea" exitCode=0 Oct 29 22:22:14.591929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.587678 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" event={"ID":"37000434-a758-4adc-8948-b541123008da","Type":"ContainerDied","Data":"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea"} Oct 29 22:22:14.591929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.587782 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" event={"ID":"37000434-a758-4adc-8948-b541123008da","Type":"ContainerDied","Data":"02ed127b757f1f2fa1026d6a026fc16414a11342eab0febf7cc1b3491998d1f1"} Oct 29 22:22:14.591929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.587817 4376 scope.go:122] "RemoveContainer" containerID="b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea" Oct 29 22:22:14.591929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.588052 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2" Oct 29 22:22:14.627461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.627440 4376 scope.go:122] "RemoveContainer" containerID="b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea" Oct 29 22:22:14.627883 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:22:14.627849 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea\": container with ID starting with b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea not found: ID does not exist" containerID="b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea" Oct 29 22:22:14.628029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.627898 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea"} err="failed to get container status \"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea\": rpc error: code = NotFound desc = could not find container \"b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea\": container with ID starting with b058509dfdcda63e863e47ce76ae62efad0f233ddd32845c0bba16705dbd24ea not found: ID does not exist" Oct 29 22:22:14.667978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.667464 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:22:14.674497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:14.674445 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4rzq2"] Oct 29 22:22:15.277900 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:15.277856 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37000434-a758-4adc-8948-b541123008da" path="/var/lib/kubelet/pods/37000434-a758-4adc-8948-b541123008da/volumes" Oct 29 22:22:16.438987 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:22:16.438951 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:22:17.618534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.618501 4376 generic.go:356] "Generic (PLEG): container finished" podID="33c5c73a-3d11-4406-b921-d053f710c6da" containerID="18941031fcdc24f700a1d0abaa522fe5e7435110adeeb982aa4125ab34086b69" exitCode=0 Oct 29 22:22:17.619321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.618581 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerDied","Data":"18941031fcdc24f700a1d0abaa522fe5e7435110adeeb982aa4125ab34086b69"} Oct 29 22:22:17.619321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.619129 4376 scope.go:122] "RemoveContainer" containerID="18941031fcdc24f700a1d0abaa522fe5e7435110adeeb982aa4125ab34086b69" Oct 29 22:22:17.620500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.620476 4376 generic.go:356] "Generic (PLEG): container finished" podID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerID="2d4cde3911e106537ab83dfbd84748ade7c4e51f1898a33300683f24ca837bbc" exitCode=0 Oct 29 22:22:17.620629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.620505 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerDied","Data":"2d4cde3911e106537ab83dfbd84748ade7c4e51f1898a33300683f24ca837bbc"} Oct 29 22:22:17.620896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.620881 4376 scope.go:122] "RemoveContainer" containerID="2d4cde3911e106537ab83dfbd84748ade7c4e51f1898a33300683f24ca837bbc" Oct 29 22:22:17.621300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.621268 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:17.621392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.621361 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:17.621437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.621391 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerName="main" containerID="cri-o://e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592" gracePeriod=45 Oct 29 22:22:17.621538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.621432 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:17.622786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.622758 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:17.622896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.622833 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:17.622896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:17.622869 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:18.172092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.169968 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:22:18.172092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.171308 4376 cpu_manager.go:414] "containerMap: removing container" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" Oct 29 22:22:18.172092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.171334 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" Oct 29 22:22:18.172092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.171527 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="37000434-a758-4adc-8948-b541123008da" containerName="istio-proxy" Oct 29 22:22:18.189935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.188782 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.255242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.208351 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c1015-openshift-default-dockercfg-pq5m2\"" Oct 29 22:22:18.336943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.334368 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:22:18.352177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352145 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352347 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c727g\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352491 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352597 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352646 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352706 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.352793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.352766 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.463975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.463682 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.464228 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.464281 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.464955 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.465756 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c727g\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.465940 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.466145 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.466204 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.466260 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.466315 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.467494 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.468044 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.468324 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.470925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.470258 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.485223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.478371 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:22:18.499596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.499563 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c727g\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.503581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.503417 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.504102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.503709 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.504102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.504032 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\") pod \"e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.568069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568035 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.568264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568118 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.568264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568222 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.568385 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568311 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.568438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568402 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.568578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.568561 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\" (UniqueName: \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\") pod \"05e1d894-35f0-419e-b2e6-cca6624024d8\" (UID: \"05e1d894-35f0-419e-b2e6-cca6624024d8\") " Oct 29 22:22:18.569468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.569098 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:18.569468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.569230 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:18.569895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.569874 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:18.570033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.570006 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.570159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.570148 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.572346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.572127 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:18.572346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.572289 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "kube-api-access-lfqlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:18.572810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.572779 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:18.573838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.573813 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs" pod "05e1d894-35f0-419e-b2e6-cca6624024d8" (UID: "05e1d894-35f0-419e-b2e6-cca6624024d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:22:18.631615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.631312 4376 generic.go:356] "Generic (PLEG): container finished" podID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerID="e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592" exitCode=0 Oct 29 22:22:18.635925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.632934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" event={"ID":"05e1d894-35f0-419e-b2e6-cca6624024d8","Type":"ContainerDied","Data":"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592"} Oct 29 22:22:18.635925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.632987 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" event={"ID":"05e1d894-35f0-419e-b2e6-cca6624024d8","Type":"ContainerDied","Data":"87bff84bb67c5ec43dc957abf8bfd614cd6f3b70b5d1cd1c7edb93803f80d1fb"} Oct 29 22:22:18.635925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.633017 4376 scope.go:122] "RemoveContainer" containerID="e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592" Oct 29 22:22:18.635925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.633141 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" Oct 29 22:22:18.637995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.637581 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerStarted","Data":"bc2bb2b8b66af7226d0fb9ebaa27ae8859619c339f476eca5def407b2cc26231"} Oct 29 22:22:18.640344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.640290 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerStarted","Data":"6da5a7c83029cb0cc3b7e8c4f65e4a1823c04643e75fc4e200cdd5d6a1fd5aa6"} Oct 29 22:22:18.661153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.661124 4376 scope.go:122] "RemoveContainer" containerID="e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592" Oct 29 22:22:18.662741 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:22:18.661732 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592\": container with ID starting with e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592 not found: ID does not exist" containerID="e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592" Oct 29 22:22:18.662741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.661771 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592"} err="failed to get container status \"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592\": rpc error: code = NotFound desc = could not find container \"e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592\": container with ID starting with e47af5145c53c69e4fc9f48c11cc377a6c518fa960bb142ecb96adf8d644c592 not found: ID does not exist" Oct 29 22:22:18.672616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.672528 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqlj\" (UniqueName: \"kubernetes.io/projected/05e1d894-35f0-419e-b2e6-cca6624024d8-kube-api-access-lfqlj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.672616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.672561 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.672616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.672578 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e1d894-35f0-419e-b2e6-cca6624024d8-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.672616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.672593 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e1d894-35f0-419e-b2e6-cca6624024d8-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:18.746596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.746518 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:22:18.748724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.748703 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659"] Oct 29 22:22:18.928228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.922494 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:22:18.930891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.930851 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:18.931023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.930983 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:18.931023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:18.931016 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:19.003824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.003733 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:22:19.005492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.005467 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:19.005618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.005510 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:19.057657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.056645 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:22:19.058067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.057939 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:19.058067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.058010 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:19.126706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.126662 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bef83b/auth-test-model-bef83b-kserve-75b6ddf845-kb659" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:22:19.306119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:19.306033 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" path="/var/lib/kubelet/pods/05e1d894-35f0-419e-b2e6-cca6624024d8/volumes" Oct 29 22:22:20.005270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.005236 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:20.005856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.005819 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:20.021100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.021035 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerStarted","Data":"ae732de450232978466bb7246d639597d4fa6f841abb5bc4fe9e5dbcc4371666"} Oct 29 22:22:20.021473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.021449 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerStarted","Data":"6b09f2d8ba61ea4ef0f26eae35589dc8cc50bd17618a5d70eb62025560eaec6d"} Oct 29 22:22:20.087929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.083350 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:20.087929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.083428 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:20.574850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.574810 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:20.586429 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.586393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:20.664930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.656769 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podStartSLOduration=2.656749355 podStartE2EDuration="2.656749355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:20.085513148 +0000 UTC m=+2398.933989991" watchObservedRunningTime="2026-10-29 22:22:20.656749355 +0000 UTC m=+2399.505226201" Oct 29 22:22:20.815122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:20.815077 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:22:21.012926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.010017 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:21.012926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.010083 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:21.057200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.055765 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:21.057509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.057472 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:21.098797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.098229 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:21.217389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:21.216360 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:22:22.015094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:22.015059 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:22.015729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:22.015695 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:22.049050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:22.049005 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:22.049254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:22.049073 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:22.275093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:22.275001 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:22:23.004473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:23.004428 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:23.004759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:23.004487 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:23.053930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:23.050591 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:23.053930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:23.050657 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:23.596057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:23.595984 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:22:24.003619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.003585 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:24.003816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.003650 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:24.051930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.051009 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 29 22:22:24.051930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.051072 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 29 22:22:24.880930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.880882 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:22:24.882980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.882954 4376 cpu_manager.go:414] "containerMap: removing container" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerName="main" Oct 29 22:22:24.883190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.883176 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerName="main" Oct 29 22:22:24.883593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.883579 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e1d894-35f0-419e-b2e6-cca6624024d8" containerName="main" Oct 29 22:22:24.888457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.888335 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:24.898263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.898215 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:22:24.906505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.906479 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:24.907002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.906979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qs7\" (UniqueName: \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:24.908797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.908778 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c1015\"" Oct 29 22:22:24.917180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:24.917103 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c1015-dockercfg-67k7j\"" Oct 29 22:22:25.102599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.043751 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:22:25.102599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.043827 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:22:25.102599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.045188 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qs7\" (UniqueName: \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:25.102599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.045309 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:25.129001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.103633 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:25.218189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.218157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qs7\" (UniqueName: \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\") pod \"payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:25.341339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.341306 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:25.392159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.387341 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:22:25.392463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.392439 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:22:25.392730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.392706 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.514198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.512996 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:22:25.515468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.515414 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:22:25.601484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.601433 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7sw\" (UniqueName: \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.601661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.601530 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.703306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.703266 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7sw\" (UniqueName: \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.703519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.703369 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.704556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.704533 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.716300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.716267 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:22:25.717938 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:22:25.717876 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a19411f_562d_4d77_8a97_e9f73d1079a1.slice/crio-31eb0dae10858a7408d5a2a6dc8569367b6ba7a2bd21f26cfd1a83c55ecb3fee WatchSource:0}: Error finding container 31eb0dae10858a7408d5a2a6dc8569367b6ba7a2bd21f26cfd1a83c55ecb3fee: Status 404 returned error can't find the container with id 31eb0dae10858a7408d5a2a6dc8569367b6ba7a2bd21f26cfd1a83c55ecb3fee Oct 29 22:22:25.747854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.747817 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7sw\" (UniqueName: \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\") pod \"payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:25.749718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:25.749691 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:26.008807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.008726 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:22:26.010275 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.010250 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:22:26.070988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.070957 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:22:26.073168 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:22:26.073140 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb769b61_43aa_4b69_9504_d4a4332cbbed.slice/crio-feb8299a9e341b0de2efabbc5e40d0e6c9251cb4ecadc7526f7ad36ca57172b3 WatchSource:0}: Error finding container feb8299a9e341b0de2efabbc5e40d0e6c9251cb4ecadc7526f7ad36ca57172b3: Status 404 returned error can't find the container with id feb8299a9e341b0de2efabbc5e40d0e6c9251cb4ecadc7526f7ad36ca57172b3 Oct 29 22:22:26.098817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.098784 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" podUID="df28ada4-0968-4efd-8049-9b3fe37b31c8" containerName="payload-processing" containerID="cri-o://31167bcdd9f34f450c30d0dc6152fed60c2007d212ff48d06078fdca37b6af87" gracePeriod=25 Oct 29 22:22:26.381200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.381164 4376 generic.go:356] "Generic (PLEG): container finished" podID="df28ada4-0968-4efd-8049-9b3fe37b31c8" containerID="31167bcdd9f34f450c30d0dc6152fed60c2007d212ff48d06078fdca37b6af87" exitCode=0 Oct 29 22:22:26.381393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.381234 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" event={"ID":"df28ada4-0968-4efd-8049-9b3fe37b31c8","Type":"ContainerDied","Data":"31167bcdd9f34f450c30d0dc6152fed60c2007d212ff48d06078fdca37b6af87"} Oct 29 22:22:26.383119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.383090 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" event={"ID":"bb769b61-43aa-4b69-9504-d4a4332cbbed","Type":"ContainerStarted","Data":"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3"} Oct 29 22:22:26.383280 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.383137 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" event={"ID":"bb769b61-43aa-4b69-9504-d4a4332cbbed","Type":"ContainerStarted","Data":"feb8299a9e341b0de2efabbc5e40d0e6c9251cb4ecadc7526f7ad36ca57172b3"} Oct 29 22:22:26.383280 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.383180 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:26.385694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.385672 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" event={"ID":"5a19411f-562d-4d77-8a97-e9f73d1079a1","Type":"ContainerStarted","Data":"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250"} Oct 29 22:22:26.385816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.385718 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:26.385816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.385741 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" event={"ID":"5a19411f-562d-4d77-8a97-e9f73d1079a1","Type":"ContainerStarted","Data":"31eb0dae10858a7408d5a2a6dc8569367b6ba7a2bd21f26cfd1a83c55ecb3fee"} Oct 29 22:22:26.428181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.427704 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" podStartSLOduration=2.427686896 podStartE2EDuration="2.427686896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:26.425040114 +0000 UTC m=+2405.273516959" watchObservedRunningTime="2026-10-29 22:22:26.427686896 +0000 UTC m=+2405.276163743" Oct 29 22:22:26.429048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.429025 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:22:26.470982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.470872 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" podStartSLOduration=2.470852489 podStartE2EDuration="2.470852489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:26.469431909 +0000 UTC m=+2405.317908745" watchObservedRunningTime="2026-10-29 22:22:26.470852489 +0000 UTC m=+2405.319329335" Oct 29 22:22:26.529034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.528994 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\") pod \"df28ada4-0968-4efd-8049-9b3fe37b31c8\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " Oct 29 22:22:26.529231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.529062 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\") pod \"df28ada4-0968-4efd-8049-9b3fe37b31c8\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " Oct 29 22:22:26.529231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.529219 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\") pod \"df28ada4-0968-4efd-8049-9b3fe37b31c8\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " Oct 29 22:22:26.529650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.529617 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\") pod \"df28ada4-0968-4efd-8049-9b3fe37b31c8\" (UID: \"df28ada4-0968-4efd-8049-9b3fe37b31c8\") " Oct 29 22:22:26.529939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.529879 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume" pod "df28ada4-0968-4efd-8049-9b3fe37b31c8" (UID: "df28ada4-0968-4efd-8049-9b3fe37b31c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:26.530236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.530213 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume" pod "df28ada4-0968-4efd-8049-9b3fe37b31c8" (UID: "df28ada4-0968-4efd-8049-9b3fe37b31c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:26.532468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.532439 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:26.532567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.532473 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df28ada4-0968-4efd-8049-9b3fe37b31c8-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:26.532567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.532475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4" pod "df28ada4-0968-4efd-8049-9b3fe37b31c8" (UID: "df28ada4-0968-4efd-8049-9b3fe37b31c8"). InnerVolumeSpecName "kube-api-access-b9bk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:26.532681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.532627 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials" pod "df28ada4-0968-4efd-8049-9b3fe37b31c8" (UID: "df28ada4-0968-4efd-8049-9b3fe37b31c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:26.643983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.640484 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:26.643983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:26.640526 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bk4\" (UniqueName: \"kubernetes.io/projected/df28ada4-0968-4efd-8049-9b3fe37b31c8-kube-api-access-b9bk4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:27.317929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.314992 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:22:27.372928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.369610 4376 cpu_manager.go:414] "containerMap: removing container" podUID="df28ada4-0968-4efd-8049-9b3fe37b31c8" containerName="payload-processing" Oct 29 22:22:27.372928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.369638 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df28ada4-0968-4efd-8049-9b3fe37b31c8" containerName="payload-processing" Oct 29 22:22:27.372928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.369896 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="df28ada4-0968-4efd-8049-9b3fe37b31c8" containerName="payload-processing" Oct 29 22:22:27.434614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.434585 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" Oct 29 22:22:27.436993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.436967 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.457375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.457345 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:22:27.457491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.457416 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv" event={"ID":"df28ada4-0968-4efd-8049-9b3fe37b31c8","Type":"ContainerDied","Data":"9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9"} Oct 29 22:22:27.457491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.457463 4376 scope.go:122] "RemoveContainer" containerID="31167bcdd9f34f450c30d0dc6152fed60c2007d212ff48d06078fdca37b6af87" Oct 29 22:22:27.466290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.466255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpkz\" (UniqueName: \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.466790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.466761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.468619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.468512 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-50d4af-dockercfg-qz99g\"" Oct 29 22:22:27.468619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.468603 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-50d4af\"" Oct 29 22:22:27.515621 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:22:27.515585 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf28ada4_0968_4efd_8049_9b3fe37b31c8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf28ada4_0968_4efd_8049_9b3fe37b31c8.slice/crio-9e10e96af3b14b39483cc8d73f756db58ee7e46c463555f67734e43077baa6e9\": RecentStats: unable to find data in memory cache]" Oct 29 22:22:27.537869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.537831 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:22:27.542724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.542695 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-db44ht2gv"] Oct 29 22:22:27.568914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.568826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.569070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.568972 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpkz\" (UniqueName: \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.569488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.569466 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.597119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.597086 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpkz\" (UniqueName: \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\") pod \"payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:27.771285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:27.771250 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:28.081619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:28.081586 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:22:28.082399 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:22:28.082371 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4373ca_fcb6_49f3_8b48_b735bab0c859.slice/crio-af0105e27e6f3924a648aa03d923725848d0fb5c36f1c42d0584d10dd571b73f WatchSource:0}: Error finding container af0105e27e6f3924a648aa03d923725848d0fb5c36f1c42d0584d10dd571b73f: Status 404 returned error can't find the container with id af0105e27e6f3924a648aa03d923725848d0fb5c36f1c42d0584d10dd571b73f Oct 29 22:22:28.445152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:28.445063 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" event={"ID":"1e4373ca-fcb6-49f3-8b48-b735bab0c859","Type":"ContainerStarted","Data":"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0"} Oct 29 22:22:28.445152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:28.445122 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" event={"ID":"1e4373ca-fcb6-49f3-8b48-b735bab0c859","Type":"ContainerStarted","Data":"af0105e27e6f3924a648aa03d923725848d0fb5c36f1c42d0584d10dd571b73f"} Oct 29 22:22:28.445602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:28.445275 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:28.486809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:28.486754 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" podStartSLOduration=1.486740897 podStartE2EDuration="1.486740897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:28.486135303 +0000 UTC m=+2407.334612172" watchObservedRunningTime="2026-10-29 22:22:28.486740897 +0000 UTC m=+2407.335217742" Oct 29 22:22:29.274229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:29.274196 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df28ada4-0968-4efd-8049-9b3fe37b31c8" path="/var/lib/kubelet/pods/df28ada4-0968-4efd-8049-9b3fe37b31c8/volumes" Oct 29 22:22:37.439240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:37.439203 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:22:37.439678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:37.439291 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:22:39.456618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:39.456588 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:22:46.453124 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:22:46.453085 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:22:56.885354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.885312 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:22:56.890819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.890778 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.909880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.909853 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:22:56.912344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912315 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912558 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912613 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912667 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2w4\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912742 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912823 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.912924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.912870 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.913097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.913073 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.913447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.913230 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:56.919943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:56.917331 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-529e60-openshift-default-dockercfg-8lgd9\"" Oct 29 22:22:57.015326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015295 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015380 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015493 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015618 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015681 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015868 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015887 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015896 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2w4\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.015963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.015940 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.016253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.016230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.016253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.016246 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.016357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.016341 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.017766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.017750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.017826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.017777 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.040275 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.040255 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.040646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.040624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2w4\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\") pod \"e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.223927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.223593 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:57.784105 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:22:57.784078 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5523068f_66af_4a90_ad67_035ed3f37fb4.slice/crio-5579fa1bc5337956c1608a3c064398f1e62288ad46d88d57691412883bf440d9 WatchSource:0}: Error finding container 5579fa1bc5337956c1608a3c064398f1e62288ad46d88d57691412883bf440d9: Status 404 returned error can't find the container with id 5579fa1bc5337956c1608a3c064398f1e62288ad46d88d57691412883bf440d9 Oct 29 22:22:57.784215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.784153 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:22:57.787245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.787210 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:57.787359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.787289 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:57.787359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:57.787334 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:22:58.723038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:58.723001 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerStarted","Data":"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd"} Oct 29 22:22:58.723445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:58.723060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerStarted","Data":"5579fa1bc5337956c1608a3c064398f1e62288ad46d88d57691412883bf440d9"} Oct 29 22:22:58.796046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:58.795998 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podStartSLOduration=2.79598446 podStartE2EDuration="2.79598446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:58.782884779 +0000 UTC m=+2437.631361620" watchObservedRunningTime="2026-10-29 22:22:58.79598446 +0000 UTC m=+2437.644461305" Oct 29 22:22:59.223992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:59.223962 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:59.231293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:59.231265 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:59.741241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:59.741208 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:22:59.742431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:22:59.742399 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:23:08.274322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.274287 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:23:08.279052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.279028 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.307882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.307851 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"routing-overlay\"" Oct 29 22:23:08.308197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.308177 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"kube-root-ca.crt\"" Oct 29 22:23:08.308427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.308411 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-plugins-e2e-extmod-b-07e166\"" Oct 29 22:23:08.308603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.308589 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"openshift-service-ca.crt\"" Oct 29 22:23:08.311563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.308765 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-58ea3cdf2b14\"" Oct 29 22:23:08.312551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.312521 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:23:08.315779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.315757 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-b9b88c959337\"" Oct 29 22:23:08.316053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.316036 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-e2e--d445abcd-dockercfg-f5q7z\"" Oct 29 22:23:08.382474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.382442 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.382656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.382599 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.382764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.382744 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgx4\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.382943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.382922 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.484442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.484400 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.484656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.484524 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.484656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.484630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgx4\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.484800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.484714 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.485294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.485267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.485294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.485271 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.487410 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.487385 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.509626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.509598 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgx4\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.596919 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.596810 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:08.920393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:08.920366 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:23:08.921283 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:23:08.921258 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c65fde1_8e3e_420c_94b4_f808f33ebb5d.slice/crio-7d1a6a5fa288bdbf54ea51e807ebab9e6d9e69a1f0873e4f1e92ad862d71583d WatchSource:0}: Error finding container 7d1a6a5fa288bdbf54ea51e807ebab9e6d9e69a1f0873e4f1e92ad862d71583d: Status 404 returned error can't find the container with id 7d1a6a5fa288bdbf54ea51e807ebab9e6d9e69a1f0873e4f1e92ad862d71583d Oct 29 22:23:09.862876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:09.862842 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" event={"ID":"9c65fde1-8e3e-420c-94b4-f808f33ebb5d","Type":"ContainerStarted","Data":"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605"} Oct 29 22:23:09.863283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:09.862897 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" event={"ID":"9c65fde1-8e3e-420c-94b4-f808f33ebb5d","Type":"ContainerStarted","Data":"7d1a6a5fa288bdbf54ea51e807ebab9e6d9e69a1f0873e4f1e92ad862d71583d"} Oct 29 22:23:09.863283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:09.862954 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:09.908603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:09.908556 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" podStartSLOduration=1.908543893 podStartE2EDuration="1.908543893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:09.905219708 +0000 UTC m=+2448.753696554" watchObservedRunningTime="2026-10-29 22:23:09.908543893 +0000 UTC m=+2448.757020738" Oct 29 22:23:16.436822 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:23:16.436787 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:23:17.892075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:17.892033 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:23:20.873810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:20.873782 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:23:22.893751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:22.893708 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" podUID="11381e51-f9a6-457f-989e-6965ae2609da" containerName="payload-processing" containerID="cri-o://4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a" gracePeriod=25 Oct 29 22:23:23.171592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.171567 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:23:23.299651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.299623 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\") pod \"11381e51-f9a6-457f-989e-6965ae2609da\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " Oct 29 22:23:23.299819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.299682 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\") pod \"11381e51-f9a6-457f-989e-6965ae2609da\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " Oct 29 22:23:23.299819 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.299776 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\") pod \"11381e51-f9a6-457f-989e-6965ae2609da\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " Oct 29 22:23:23.300069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.300047 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\") pod \"11381e51-f9a6-457f-989e-6965ae2609da\" (UID: \"11381e51-f9a6-457f-989e-6965ae2609da\") " Oct 29 22:23:23.300132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.300083 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume" pod "11381e51-f9a6-457f-989e-6965ae2609da" (UID: "11381e51-f9a6-457f-989e-6965ae2609da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:23:23.300230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.300209 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume" pod "11381e51-f9a6-457f-989e-6965ae2609da" (UID: "11381e51-f9a6-457f-989e-6965ae2609da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:23:23.300655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.300638 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:23.300718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.300657 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11381e51-f9a6-457f-989e-6965ae2609da-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:23.302251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.302228 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials" pod "11381e51-f9a6-457f-989e-6965ae2609da" (UID: "11381e51-f9a6-457f-989e-6965ae2609da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:23.302350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.302259 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4" pod "11381e51-f9a6-457f-989e-6965ae2609da" (UID: "11381e51-f9a6-457f-989e-6965ae2609da"). InnerVolumeSpecName "kube-api-access-bc9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:23.402811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.402728 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc9g4\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-kube-api-access-bc9g4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:23.402811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.402759 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11381e51-f9a6-457f-989e-6965ae2609da-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:23.993146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.993111 4376 generic.go:356] "Generic (PLEG): container finished" podID="11381e51-f9a6-457f-989e-6965ae2609da" containerID="4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a" exitCode=0 Oct 29 22:23:23.993595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.993154 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" Oct 29 22:23:23.993595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.993162 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" event={"ID":"11381e51-f9a6-457f-989e-6965ae2609da","Type":"ContainerDied","Data":"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a"} Oct 29 22:23:23.993595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.993202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h" event={"ID":"11381e51-f9a6-457f-989e-6965ae2609da","Type":"ContainerDied","Data":"6bb3eb37099619f5441b4194e7802c21ca471ed39f9d623b8f56dc9801748481"} Oct 29 22:23:23.993595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:23.993223 4376 scope.go:122] "RemoveContainer" containerID="4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a" Oct 29 22:23:24.005204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:24.005187 4376 scope.go:122] "RemoveContainer" containerID="4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a" Oct 29 22:23:24.005435 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:23:24.005416 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a\": container with ID starting with 4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a not found: ID does not exist" containerID="4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a" Oct 29 22:23:24.005492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:24.005442 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a"} err="failed to get container status \"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a\": rpc error: code = NotFound desc = could not find container \"4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a\": container with ID starting with 4ab0a0b362328dbdc5bd5345b974c8a1f9fb69af2d3ac6474f0e23ecb09f8a3a not found: ID does not exist" Oct 29 22:23:24.050297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:24.050269 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:23:24.062154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:24.062124 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-68b5nhb5h"] Oct 29 22:23:25.273689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:25.273658 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11381e51-f9a6-457f-989e-6965ae2609da" path="/var/lib/kubelet/pods/11381e51-f9a6-457f-989e-6965ae2609da/volumes" Oct 29 22:23:35.620998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.620771 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:23:35.622710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.622680 4376 cpu_manager.go:414] "containerMap: removing container" podUID="11381e51-f9a6-457f-989e-6965ae2609da" containerName="payload-processing" Oct 29 22:23:35.622710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.622705 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11381e51-f9a6-457f-989e-6965ae2609da" containerName="payload-processing" Oct 29 22:23:35.622993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.622849 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="11381e51-f9a6-457f-989e-6965ae2609da" containerName="payload-processing" Oct 29 22:23:35.628256 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.627464 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.640524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.640492 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:23:35.648632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.648608 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa13069b2bcb\"" Oct 29 22:23:35.648828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.648799 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 22:23:35.649123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.649078 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 22:23:35.649252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.649156 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 22:23:35.655603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.655581 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h88vv\"" Oct 29 22:23:35.655697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.655602 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 22:23:35.756400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.756369 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.756538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.756509 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.756720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.756698 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.756760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.756731 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbpn\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.861257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.861217 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.861524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.861501 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbpn\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.861780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.861763 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.862041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.862025 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.863012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.862987 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.864621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.864219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.874005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.869648 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.899922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.897871 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbpn\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\") pod \"payload-processing-external-model-648dc5b567-pdscv\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:35.960339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:35.960293 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:36.335109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.335077 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:23:36.335544 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:23:36.335516 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50522ace_12d1_4429_b6af_ce27a8dce95e.slice/crio-a8fe923953c1cba7d6f8cdfc4fad1c3ea043f764ff2cbf74fdf62e8c7ce6d5ae WatchSource:0}: Error finding container a8fe923953c1cba7d6f8cdfc4fad1c3ea043f764ff2cbf74fdf62e8c7ce6d5ae: Status 404 returned error can't find the container with id a8fe923953c1cba7d6f8cdfc4fad1c3ea043f764ff2cbf74fdf62e8c7ce6d5ae Oct 29 22:23:36.427028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.427004 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:23:36.431179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.431150 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.449177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.448534 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-adfb9d-serving-cert\"" Oct 29 22:23:36.452749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.452723 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:23:36.587137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.587029 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.587368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.587339 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw8t\" (UniqueName: \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.587574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.587548 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.588073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.588050 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.702558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.702518 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw8t\" (UniqueName: \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.703076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.702622 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.703076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.702711 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.703076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.702770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.703076 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:23:36.702934 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-serving-cert: secret "maas-api-e2e-extmod-b-adfb9d-serving-cert" not found Oct 29 22:23:36.703076 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:23:36.703005 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls podName:6537394e-26ba-40f7-a956-1410208b0504 nodeName:}" failed. No retries permitted until 2026-10-29 22:23:37.20298666 +0000 UTC m=+2476.051463494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls") pod "maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" (UID: "6537394e-26ba-40f7-a956-1410208b0504") : secret "maas-api-e2e-extmod-b-adfb9d-serving-cert" not found Oct 29 22:23:36.703396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.703372 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.705220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.705198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:36.729127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:36.729102 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw8t\" (UniqueName: \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:37.147502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.147464 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" event={"ID":"50522ace-12d1-4429-b6af-ce27a8dce95e","Type":"ContainerStarted","Data":"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a"} Oct 29 22:23:37.147712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.147522 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" event={"ID":"50522ace-12d1-4429-b6af-ce27a8dce95e","Type":"ContainerStarted","Data":"a8fe923953c1cba7d6f8cdfc4fad1c3ea043f764ff2cbf74fdf62e8c7ce6d5ae"} Oct 29 22:23:37.147712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.147544 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:37.193017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.192970 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" podStartSLOduration=2.192957196 podStartE2EDuration="2.192957196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:37.189610667 +0000 UTC m=+2476.038087513" watchObservedRunningTime="2026-10-29 22:23:37.192957196 +0000 UTC m=+2476.041434090" Oct 29 22:23:37.212030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.212002 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:37.214375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.214348 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") pod \"maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:37.350278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.350237 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:37.697827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.697803 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:23:37.700372 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:23:37.700341 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6537394e_26ba_40f7_a956_1410208b0504.slice/crio-d70a64f9c8a4babc228e164480c40f10408f7958a6f19658b83f8afd41e76040 WatchSource:0}: Error finding container d70a64f9c8a4babc228e164480c40f10408f7958a6f19658b83f8afd41e76040: Status 404 returned error can't find the container with id d70a64f9c8a4babc228e164480c40f10408f7958a6f19658b83f8afd41e76040 Oct 29 22:23:37.702008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:37.701886 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:23:38.156524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:38.156481 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" event={"ID":"6537394e-26ba-40f7-a956-1410208b0504","Type":"ContainerStarted","Data":"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77"} Oct 29 22:23:38.156933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:38.156569 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" event={"ID":"6537394e-26ba-40f7-a956-1410208b0504","Type":"ContainerStarted","Data":"d70a64f9c8a4babc228e164480c40f10408f7958a6f19658b83f8afd41e76040"} Oct 29 22:23:38.156933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:38.156794 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:38.210948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:38.210874 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" podStartSLOduration=1.889781142 podStartE2EDuration="2.210859688s" totalImagesPullingTime="321.078546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:38.206555082 +0000 UTC m=+2477.055031949" watchObservedRunningTime="2026-10-29 22:23:38.210859688 +0000 UTC m=+2477.059336533" Oct 29 22:23:44.168471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:44.168443 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:23:46.434030 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:23:46.433993 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:23:48.160090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:48.160064 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:23:57.172197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.172155 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:23:57.177330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.177303 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.200176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.200143 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:23:57.203578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.203552 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-529e60-serving-cert\"" Oct 29 22:23:57.216570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.216528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.216829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.216804 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c944\" (UniqueName: \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.216981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.216960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.217057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.217010 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.318690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.318651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c944\" (UniqueName: \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.318948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.318742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.318948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.318779 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.319118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.319037 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.319689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.319665 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.321346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.321324 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.321443 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.321345 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.362637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.362570 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c944\" (UniqueName: \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\") pod \"maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:57.493422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:57.493374 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:58.384615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:58.380377 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:23:58.398491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:58.398453 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" event={"ID":"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9","Type":"ContainerStarted","Data":"59b445f0a41e8db5fbeb93412d0a5e3c2f68b8b0e85e7829f5e938c5409698cf"} Oct 29 22:23:59.646926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:59.643523 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" event={"ID":"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9","Type":"ContainerStarted","Data":"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb"} Oct 29 22:23:59.654929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:59.649065 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:23:59.751929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:23:59.712992 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" podStartSLOduration=2.054938104 podStartE2EDuration="2.712973631s" totalImagesPullingTime="658.035527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:59.712358963 +0000 UTC m=+2498.560835812" watchObservedRunningTime="2026-10-29 22:23:59.712973631 +0000 UTC m=+2498.561450476" Oct 29 22:24:04.713346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.713314 4376 generic.go:356] "Generic (PLEG): container finished" podID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerID="ae732de450232978466bb7246d639597d4fa6f841abb5bc4fe9e5dbcc4371666" exitCode=0 Oct 29 22:24:04.713808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.713385 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerDied","Data":"ae732de450232978466bb7246d639597d4fa6f841abb5bc4fe9e5dbcc4371666"} Oct 29 22:24:04.713808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.713804 4376 scope.go:122] "RemoveContainer" containerID="ae732de450232978466bb7246d639597d4fa6f841abb5bc4fe9e5dbcc4371666" Oct 29 22:24:04.715924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.715879 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:04.715994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.715974 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:04.716036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:04.716009 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:05.729690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:05.727940 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerStarted","Data":"ff1996bfd3bee40654da77590fadeb6ababb5f499c0cc452fa9171b7a791e99e"} Oct 29 22:24:06.109169 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:06.109027 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:24:06.572481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:06.572448 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:24:06.573942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:06.573872 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:24:06.574082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:06.573949 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:24:06.665792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:06.665766 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:24:07.573494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:07.573456 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:24:07.573975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:07.573515 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:24:08.573481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:08.573447 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:24:08.573669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:08.573501 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:24:09.573394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:09.573352 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:24:09.573587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:09.573416 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:24:10.572859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:10.572824 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:24:10.573296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:10.572888 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:24:11.577969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:11.577932 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:24:11.579123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:11.579101 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:24:13.077141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:13.077107 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:24:16.434262 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:24:16.434226 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:24:18.077941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.077878 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" podUID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" containerName="payload-processing" containerID="cri-o://bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605" gracePeriod=25 Oct 29 22:24:18.361605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.361581 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:24:18.518524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.517790 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\") pod \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " Oct 29 22:24:18.518763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.518711 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\") pod \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " Oct 29 22:24:18.519416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.519348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume" pod "9c65fde1-8e3e-420c-94b4-f808f33ebb5d" (UID: "9c65fde1-8e3e-420c-94b4-f808f33ebb5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:18.520324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.519458 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\") pod \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " Oct 29 22:24:18.520324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.519492 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\") pod \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\" (UID: \"9c65fde1-8e3e-420c-94b4-f808f33ebb5d\") " Oct 29 22:24:18.521305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.521280 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:18.521791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.521756 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume" pod "9c65fde1-8e3e-420c-94b4-f808f33ebb5d" (UID: "9c65fde1-8e3e-420c-94b4-f808f33ebb5d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:18.523332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.523310 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4" pod "9c65fde1-8e3e-420c-94b4-f808f33ebb5d" (UID: "9c65fde1-8e3e-420c-94b4-f808f33ebb5d"). InnerVolumeSpecName "kube-api-access-flgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:18.523417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.523348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials" pod "9c65fde1-8e3e-420c-94b4-f808f33ebb5d" (UID: "9c65fde1-8e3e-420c-94b4-f808f33ebb5d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:18.622871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.622800 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:18.622871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.622829 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:18.622871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.622844 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgx4\" (UniqueName: \"kubernetes.io/projected/9c65fde1-8e3e-420c-94b4-f808f33ebb5d-kube-api-access-flgx4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:18.929614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.929536 4376 generic.go:356] "Generic (PLEG): container finished" podID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" containerID="bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605" exitCode=0 Oct 29 22:24:18.929614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.929575 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" Oct 29 22:24:18.929824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.929610 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" event={"ID":"9c65fde1-8e3e-420c-94b4-f808f33ebb5d","Type":"ContainerDied","Data":"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605"} Oct 29 22:24:18.929824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.929662 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88" event={"ID":"9c65fde1-8e3e-420c-94b4-f808f33ebb5d","Type":"ContainerDied","Data":"7d1a6a5fa288bdbf54ea51e807ebab9e6d9e69a1f0873e4f1e92ad862d71583d"} Oct 29 22:24:18.929824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.929683 4376 scope.go:122] "RemoveContainer" containerID="bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605" Oct 29 22:24:18.941302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.941282 4376 scope.go:122] "RemoveContainer" containerID="bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605" Oct 29 22:24:18.941592 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:24:18.941573 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605\": container with ID starting with bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605 not found: ID does not exist" containerID="bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605" Oct 29 22:24:18.941665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.941606 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605"} err="failed to get container status \"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605\": rpc error: code = NotFound desc = could not find container \"bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605\": container with ID starting with bde28120f09badcb1bcdf998470eb204e1272e59a0ec2779fef522f836bfd605 not found: ID does not exist" Oct 29 22:24:18.990283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.990265 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:24:18.994685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:18.994667 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888f5r88"] Oct 29 22:24:19.274026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:19.273993 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" path="/var/lib/kubelet/pods/9c65fde1-8e3e-420c-94b4-f808f33ebb5d/volumes" Oct 29 22:24:42.275607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.275571 4376 generic.go:356] "Generic (PLEG): container finished" podID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerID="acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd" exitCode=0 Oct 29 22:24:42.276348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.275647 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerDied","Data":"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd"} Oct 29 22:24:42.276348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.276207 4376 scope.go:122] "RemoveContainer" containerID="acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd" Oct 29 22:24:42.278364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.278324 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:42.278482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.278405 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:42.278482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:42.278438 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:43.290163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:43.290130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerStarted","Data":"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc"} Oct 29 22:24:44.224447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:44.224405 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:24:44.226013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:44.225983 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:44.226142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:44.226040 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:44.751928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:44.745025 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:24:45.224711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:45.224675 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:45.224945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:45.224739 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:46.225239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:46.225204 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:46.225642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:46.225261 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:46.367781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:46.362882 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:24:46.468498 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:24:46.468453 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:24:47.224345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:47.224311 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:47.224535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:47.224368 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:48.224212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:48.224174 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:48.224730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:48.224234 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:49.231926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:49.231829 4376 patch_prober.go:29] interesting pod/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:24:49.231926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:49.231896 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:24:50.231063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:50.231029 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:24:50.232667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:50.232636 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:24:51.363672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.363605 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" podUID="50522ace-12d1-4429-b6af-ce27a8dce95e" containerName="payload-processing" containerID="cri-o://bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a" gracePeriod=25 Oct 29 22:24:51.657035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.657005 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:24:51.693834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.693799 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\") pod \"50522ace-12d1-4429-b6af-ce27a8dce95e\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " Oct 29 22:24:51.694068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.693851 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\") pod \"50522ace-12d1-4429-b6af-ce27a8dce95e\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " Oct 29 22:24:51.694068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.693972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\") pod \"50522ace-12d1-4429-b6af-ce27a8dce95e\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " Oct 29 22:24:51.694068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.694053 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\") pod \"50522ace-12d1-4429-b6af-ce27a8dce95e\" (UID: \"50522ace-12d1-4429-b6af-ce27a8dce95e\") " Oct 29 22:24:51.694292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.694270 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume" pod "50522ace-12d1-4429-b6af-ce27a8dce95e" (UID: "50522ace-12d1-4429-b6af-ce27a8dce95e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:51.694351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.694312 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume" pod "50522ace-12d1-4429-b6af-ce27a8dce95e" (UID: "50522ace-12d1-4429-b6af-ce27a8dce95e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:51.694708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.694692 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.694750 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.694711 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50522ace-12d1-4429-b6af-ce27a8dce95e-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.696109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.696088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials" pod "50522ace-12d1-4429-b6af-ce27a8dce95e" (UID: "50522ace-12d1-4429-b6af-ce27a8dce95e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:51.696221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.696201 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn" pod "50522ace-12d1-4429-b6af-ce27a8dce95e" (UID: "50522ace-12d1-4429-b6af-ce27a8dce95e"). InnerVolumeSpecName "kube-api-access-6dbpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:51.795946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.795899 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.795946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:51.795943 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dbpn\" (UniqueName: \"kubernetes.io/projected/50522ace-12d1-4429-b6af-ce27a8dce95e-kube-api-access-6dbpn\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.421692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.421653 4376 generic.go:356] "Generic (PLEG): container finished" podID="50522ace-12d1-4429-b6af-ce27a8dce95e" containerID="bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a" exitCode=0 Oct 29 22:24:52.422119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.421740 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" Oct 29 22:24:52.422119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.421737 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" event={"ID":"50522ace-12d1-4429-b6af-ce27a8dce95e","Type":"ContainerDied","Data":"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a"} Oct 29 22:24:52.422119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.421859 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-648dc5b567-pdscv" event={"ID":"50522ace-12d1-4429-b6af-ce27a8dce95e","Type":"ContainerDied","Data":"a8fe923953c1cba7d6f8cdfc4fad1c3ea043f764ff2cbf74fdf62e8c7ce6d5ae"} Oct 29 22:24:52.422119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.421880 4376 scope.go:122] "RemoveContainer" containerID="bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a" Oct 29 22:24:52.433100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.433081 4376 scope.go:122] "RemoveContainer" containerID="bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a" Oct 29 22:24:52.433350 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:24:52.433328 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a\": container with ID starting with bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a not found: ID does not exist" containerID="bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a" Oct 29 22:24:52.433397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.433358 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a"} err="failed to get container status \"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a\": rpc error: code = NotFound desc = could not find container \"bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a\": container with ID starting with bac3907d52f517d526e4f5ff64b031e936eccfd7887d2b214f2c73892a9a2e0a not found: ID does not exist" Oct 29 22:24:52.486421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.486381 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:24:52.495045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:52.495012 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-648dc5b567-pdscv"] Oct 29 22:24:53.274276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:53.274238 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50522ace-12d1-4429-b6af-ce27a8dce95e" path="/var/lib/kubelet/pods/50522ace-12d1-4429-b6af-ce27a8dce95e/volumes" Oct 29 22:24:57.822929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.822878 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:24:57.825354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.825318 4376 cpu_manager.go:414] "containerMap: removing container" podUID="50522ace-12d1-4429-b6af-ce27a8dce95e" containerName="payload-processing" Oct 29 22:24:57.825354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.825349 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50522ace-12d1-4429-b6af-ce27a8dce95e" containerName="payload-processing" Oct 29 22:24:57.825571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.825386 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" containerName="payload-processing" Oct 29 22:24:57.825571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.825395 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" containerName="payload-processing" Oct 29 22:24:57.825932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.825893 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="50522ace-12d1-4429-b6af-ce27a8dce95e" containerName="payload-processing" Oct 29 22:24:57.826132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.826112 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c65fde1-8e3e-420c-94b4-f808f33ebb5d" containerName="payload-processing" Oct 29 22:24:57.833742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.833711 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.851378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.851345 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:24:57.940158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940127 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940165 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940227 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4gt\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940351 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940402 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940612 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940640 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:57.940763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:57.940751 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042354 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042391 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042467 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042549 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042591 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042616 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4gt\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042660 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.042828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042707 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.043023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042852 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.043023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.043023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.042992 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.043195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.043173 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.043264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.043248 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.044964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.044943 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.045213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.045193 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.069592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.069553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.069753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.069678 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4gt\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\") pod \"maas-default-gateway-openshift-default-6f54779779-c7lfl\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.154119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.154032 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:24:58.487263 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:24:58.487220 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e70e49_a950_43e9_b72f_7a8f2afa4200.slice/crio-f6e110748d3c30eeb19332d93ce63e7492d2998576c0db56f8df93941d099fc2 WatchSource:0}: Error finding container f6e110748d3c30eeb19332d93ce63e7492d2998576c0db56f8df93941d099fc2: Status 404 returned error can't find the container with id f6e110748d3c30eeb19332d93ce63e7492d2998576c0db56f8df93941d099fc2 Oct 29 22:24:58.487439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.487352 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:24:58.492026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.491976 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:58.492214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.492077 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:58.492214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:58.492114 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:24:59.513950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:59.510021 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" event={"ID":"53e70e49-a950-43e9-b72f-7a8f2afa4200","Type":"ContainerStarted","Data":"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0"} Oct 29 22:24:59.513950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:24:59.510092 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" event={"ID":"53e70e49-a950-43e9-b72f-7a8f2afa4200","Type":"ContainerStarted","Data":"f6e110748d3c30eeb19332d93ce63e7492d2998576c0db56f8df93941d099fc2"} Oct 29 22:25:00.164047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:00.164010 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:25:00.174068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:00.174034 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:00.175096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:00.175063 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:01.155158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:01.155120 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:01.160577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:01.160090 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:02.157817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:02.157781 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:02.158478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:02.158436 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:03.158396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:03.158332 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:03.158856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:03.158394 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:04.175817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:04.175294 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:04.181604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:04.177590 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:05.170928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:05.157444 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:05.170928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:05.157502 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:06.155102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:06.155058 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:06.157194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:06.155136 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:07.155267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:07.155220 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c7lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 29 22:25:07.155681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:07.155289 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 29 22:25:09.154992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:09.154961 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:25:09.155544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:09.155525 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:25:09.156362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:09.156340 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:25:09.223274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:09.223212 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podStartSLOduration=12.223197252 podStartE2EDuration="12.223197252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:59.578255585 +0000 UTC m=+2558.426732431" watchObservedRunningTime="2026-10-29 22:25:09.223197252 +0000 UTC m=+2568.071674146" Oct 29 22:25:12.110924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.110874 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:12.117289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.117263 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.137316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.137285 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:12.144118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.144034 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"routing-overlay\"" Oct 29 22:25:12.144118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.144080 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"kube-root-ca.crt\"" Oct 29 22:25:12.144118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.144091 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"openshift-service-ca.crt\"" Oct 29 22:25:12.144381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.144138 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-31abe10e24c1\"" Oct 29 22:25:12.144381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.144209 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:25:12.146504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.146468 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-e2e--b7017604-dockercfg-gkpdl\"" Oct 29 22:25:12.273719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.273682 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.273966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.273736 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.273966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.273925 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.274284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.274256 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cl5\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.377349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.377247 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.386831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.386793 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.408141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.408106 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.409698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.409670 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cl5\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.410402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.409258 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.410564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.387804 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.418640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.418546 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.467027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.453898 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cl5\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.741915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.741874 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:12.876042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.875993 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:25:12.882892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.882869 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.904128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.904101 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:25:12.911939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.911876 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:25:12.917502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.917467 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:12.929626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.929597 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.929732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.929636 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.929818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.929788 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.929973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.929883 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.930121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.930102 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.930197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.930152 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.930316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.930293 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmdh\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.930391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.930372 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.930453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.930430 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:12.944638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:12.944610 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:25:13.031691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031598 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.031691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sg5b\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.031895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031731 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.031895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031842 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.031895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031862 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.031895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031880 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031935 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031957 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.031995 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032072 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032115 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032159 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032211 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032235 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032310 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032367 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmdh\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032421 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032452 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032478 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.032649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032518 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.032834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.032774 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.033115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.033093 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.034769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.034742 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.034894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.034879 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.058379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.058348 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.058636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.058613 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmdh\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\") pod \"maas-default-gateway-openshift-default-6f54779779-z7ql8\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.063803 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:25:13.063771 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97ceb3f_3df7_4f11_827e_ff1392bb00f7.slice/crio-df590edadec5512bbdd7e78ade8d4875d0d8af0c39d5912693820d51bc6cfbc4 WatchSource:0}: Error finding container df590edadec5512bbdd7e78ade8d4875d0d8af0c39d5912693820d51bc6cfbc4: Status 404 returned error can't find the container with id df590edadec5512bbdd7e78ade8d4875d0d8af0c39d5912693820d51bc6cfbc4 Oct 29 22:25:13.064835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.064779 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.148755 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.148916 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.148986 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149064 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149182 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sg5b\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149274 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149348 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.149894 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.170314 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.170621 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.172132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.171383 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.192945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.182564 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.220926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.213593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.220926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.214668 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:13.223290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.223198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.223477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.223446 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.226974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.226571 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sg5b\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\") pod \"maas-default-gateway-openshift-default-6f54779779-wmtjb\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.242989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.237247 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:13.348948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.346063 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" event={"ID":"c97ceb3f-3df7-4f11-827e-ff1392bb00f7","Type":"ContainerStarted","Data":"df590edadec5512bbdd7e78ade8d4875d0d8af0c39d5912693820d51bc6cfbc4"} Oct 29 22:25:13.775947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.771104 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:25:13.775947 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:25:13.773091 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5f92d3_8c87_4b8d_b443_d2d788fb2957.slice/crio-89c02eb57a4423f93a47a285de3710561365904709d42ef337e93850942b8c41 WatchSource:0}: Error finding container 89c02eb57a4423f93a47a285de3710561365904709d42ef337e93850942b8c41: Status 404 returned error can't find the container with id 89c02eb57a4423f93a47a285de3710561365904709d42ef337e93850942b8c41 Oct 29 22:25:13.779790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.778559 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:13.779790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.778651 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:13.779790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.778695 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:13.821985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.816203 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:25:13.822287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.822092 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:13.822287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.822160 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:13.822287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:13.822220 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:25:14.597197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.597140 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" event={"ID":"0a199270-753e-4d4e-8e66-306cec47549d","Type":"ContainerStarted","Data":"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64"} Oct 29 22:25:14.605197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.605112 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" event={"ID":"0a199270-753e-4d4e-8e66-306cec47549d","Type":"ContainerStarted","Data":"29b64b949727f2f006dbd8f79d8c43af1766f44d659d040105c4fc819b37bca6"} Oct 29 22:25:14.647955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.647880 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" event={"ID":"dc5f92d3-8c87-4b8d-b443-d2d788fb2957","Type":"ContainerStarted","Data":"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047"} Oct 29 22:25:14.651384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.651353 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" event={"ID":"dc5f92d3-8c87-4b8d-b443-d2d788fb2957","Type":"ContainerStarted","Data":"89c02eb57a4423f93a47a285de3710561365904709d42ef337e93850942b8c41"} Oct 29 22:25:14.653005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.652972 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" event={"ID":"c97ceb3f-3df7-4f11-827e-ff1392bb00f7","Type":"ContainerStarted","Data":"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b"} Oct 29 22:25:14.654777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.654443 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:14.671276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.671234 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podStartSLOduration=2.671217428 podStartE2EDuration="2.671217428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:14.655518739 +0000 UTC m=+2573.503995584" watchObservedRunningTime="2026-10-29 22:25:14.671217428 +0000 UTC m=+2573.519694274" Oct 29 22:25:14.709873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.709812 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" podStartSLOduration=2.709792463 podStartE2EDuration="2.709792463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:14.705102902 +0000 UTC m=+2573.553579746" watchObservedRunningTime="2026-10-29 22:25:14.709792463 +0000 UTC m=+2573.558269356" Oct 29 22:25:14.758674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:14.758616 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podStartSLOduration=2.7585972 podStartE2EDuration="2.7585972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:14.756817689 +0000 UTC m=+2573.605294533" watchObservedRunningTime="2026-10-29 22:25:14.7585972 +0000 UTC m=+2573.607074047" Oct 29 22:25:15.216097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.216061 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:15.217358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.217329 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:15.217526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.217401 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:15.253347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.241494 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:15.255934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.254197 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:15.255934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:15.254266 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:16.215837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:16.215802 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:16.216255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:16.215883 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:16.253930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:16.243166 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:16.253930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:16.243220 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:16.482978 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:16.482377 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:25:17.226924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:17.226038 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:17.226924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:17.226102 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:17.268926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:17.268534 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:17.268926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:17.268615 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:18.215386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:18.215348 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:18.215583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:18.215403 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:18.250953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:18.250888 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:18.282335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:18.282276 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:19.215714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:19.215674 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:19.215897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:19.215746 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:19.265927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:19.265017 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:19.265927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:19.265095 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:19.996361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:19.996330 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:25:20.002219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.002192 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.004889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.004859 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:25:20.032639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.032600 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-e2e--b64eb102-dockercfg-2z6nl\"" Oct 29 22:25:20.032798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.032726 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"routing-overlay\"" Oct 29 22:25:20.032798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.032771 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"kube-root-ca.crt\"" Oct 29 22:25:20.049044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.032851 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-adfb9d\"" Oct 29 22:25:20.049044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.032957 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"openshift-service-ca.crt\"" Oct 29 22:25:20.049044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.033030 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"agc-cred-6cfe15a5e916\"" Oct 29 22:25:20.252802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.221663 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:20.252802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.221853 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:20.252802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.240753 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:20.252802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.240819 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:20.270685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.258851 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.270685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.259075 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.270685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.259175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.270685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.259389 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkf7\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.365960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.362753 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkf7\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.365960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.362866 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.365960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.363064 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.365960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.363164 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.365960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.364357 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.384831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.383722 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.441927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.440956 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.446611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.446535 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkf7\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:20.623985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:20.621229 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:21.250313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.240436 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:21.250313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.240486 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:21.273784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.273745 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:21.274363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.273817 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:21.526899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.526871 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:25:21.942646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.942549 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" event={"ID":"aa37120d-3fbe-4fe5-b488-247d10a8fb3c","Type":"ContainerStarted","Data":"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82"} Oct 29 22:25:21.942646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.942620 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" event={"ID":"aa37120d-3fbe-4fe5-b488-247d10a8fb3c","Type":"ContainerStarted","Data":"b3464f1d95210b4e61181e16689f9ad501b37749c158f865274d8370f6a480fa"} Oct 29 22:25:21.988367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:21.988313 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" podStartSLOduration=2.988297592 podStartE2EDuration="2.988297592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:21.988139088 +0000 UTC m=+2580.836615933" watchObservedRunningTime="2026-10-29 22:25:21.988297592 +0000 UTC m=+2580.836774437" Oct 29 22:25:22.220041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:22.217046 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:22.220041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:22.217124 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:22.241448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:22.241413 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:22.241660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:22.241461 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:22.950834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:22.950799 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:23.216347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:23.215835 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-z7ql8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:25:23.216347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:23.215896 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:25:23.241001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:23.240954 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-wmtjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:25:23.241227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:23.241029 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 29 22:25:24.717484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.717443 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:24.718240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.718213 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:24.719022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.718989 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:25:24.792434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.792398 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:24.793154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.793004 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:24.794464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:24.794441 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:25:26.725647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:26.725621 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:34.062376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:34.062338 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:25:34.869624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:34.869585 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:39.359476 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.359439 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:25:39.366377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.366348 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.377548 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.377514 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.377700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.377571 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.377785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.377772 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.377859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.377846 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7przv\" (UniqueName: \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.385509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.385481 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-797e0c-serving-cert\"" Oct 29 22:25:39.387504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.387479 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:25:39.479071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.479029 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.479071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.479072 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.479307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.479190 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.479307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.479233 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7przv\" (UniqueName: \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.479307 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:39.479243 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-serving-cert: secret "maas-api-e2e-extmod-b-797e0c-serving-cert" not found Oct 29 22:25:39.479438 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:39.479316 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls podName:db0c730c-60b9-4401-b27b-3f8459333391 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:39.979295813 +0000 UTC m=+2598.827772655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls") pod "maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" (UID: "db0c730c-60b9-4401-b27b-3f8459333391") : secret "maas-api-e2e-extmod-b-797e0c-serving-cert" not found Oct 29 22:25:39.479855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.479834 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.482063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.482040 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.506374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.506342 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7przv\" (UniqueName: \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.870638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.870594 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" podUID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" containerName="payload-processing" containerID="cri-o://b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b" gracePeriod=25 Oct 29 22:25:39.988268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.988209 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:39.990502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:39.990479 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") pod \"maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:40.159634 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.159608 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:40.165202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.165173 4376 generic.go:356] "Generic (PLEG): container finished" podID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" containerID="b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b" exitCode=0 Oct 29 22:25:40.165330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.165216 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" Oct 29 22:25:40.165330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.165254 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" event={"ID":"c97ceb3f-3df7-4f11-827e-ff1392bb00f7","Type":"ContainerDied","Data":"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b"} Oct 29 22:25:40.165330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.165312 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx" event={"ID":"c97ceb3f-3df7-4f11-827e-ff1392bb00f7","Type":"ContainerDied","Data":"df590edadec5512bbdd7e78ade8d4875d0d8af0c39d5912693820d51bc6cfbc4"} Oct 29 22:25:40.165439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.165333 4376 scope.go:122] "RemoveContainer" containerID="b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b" Oct 29 22:25:40.175732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.175715 4376 scope.go:122] "RemoveContainer" containerID="b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b" Oct 29 22:25:40.176040 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:40.176017 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b\": container with ID starting with b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b not found: ID does not exist" containerID="b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b" Oct 29 22:25:40.176151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.176050 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b"} err="failed to get container status \"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b\": rpc error: code = NotFound desc = could not find container \"b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b\": container with ID starting with b60184174e915162b389d864a6e13d2288de9f98f2e9535b208ade4caaae9d1b not found: ID does not exist" Oct 29 22:25:40.191266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.191238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\") pod \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " Oct 29 22:25:40.191519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.191500 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\") pod \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " Oct 29 22:25:40.191598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.191573 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\") pod \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " Oct 29 22:25:40.191715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.191696 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\") pod \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\" (UID: \"c97ceb3f-3df7-4f11-827e-ff1392bb00f7\") " Oct 29 22:25:40.192240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.191940 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume" pod "c97ceb3f-3df7-4f11-827e-ff1392bb00f7" (UID: "c97ceb3f-3df7-4f11-827e-ff1392bb00f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:40.192240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.192063 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume" pod "c97ceb3f-3df7-4f11-827e-ff1392bb00f7" (UID: "c97ceb3f-3df7-4f11-827e-ff1392bb00f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:40.192579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.192546 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:40.192579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.192575 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:40.193834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.193808 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials" pod "c97ceb3f-3df7-4f11-827e-ff1392bb00f7" (UID: "c97ceb3f-3df7-4f11-827e-ff1392bb00f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:40.193976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.193925 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5" pod "c97ceb3f-3df7-4f11-827e-ff1392bb00f7" (UID: "c97ceb3f-3df7-4f11-827e-ff1392bb00f7"). InnerVolumeSpecName "kube-api-access-67cl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:40.288689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.287508 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:40.294322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.294293 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67cl5\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-kube-api-access-67cl5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:40.294322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.294321 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c97ceb3f-3df7-4f11-827e-ff1392bb00f7-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:40.593935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.586798 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:40.593935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.592719 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdsn9rx"] Oct 29 22:25:40.864964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:40.864923 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:25:40.938948 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:25:40.938360 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0c730c_60b9_4401_b27b_3f8459333391.slice/crio-8aa680410295a402d59615ada4b150e4069373feec7784b2e892f0276a2386b5 WatchSource:0}: Error finding container 8aa680410295a402d59615ada4b150e4069373feec7784b2e892f0276a2386b5: Status 404 returned error can't find the container with id 8aa680410295a402d59615ada4b150e4069373feec7784b2e892f0276a2386b5 Oct 29 22:25:41.176486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:41.176408 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" event={"ID":"db0c730c-60b9-4401-b27b-3f8459333391","Type":"ContainerStarted","Data":"8aa680410295a402d59615ada4b150e4069373feec7784b2e892f0276a2386b5"} Oct 29 22:25:41.279297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:41.279244 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" path="/var/lib/kubelet/pods/c97ceb3f-3df7-4f11-827e-ff1392bb00f7/volumes" Oct 29 22:25:42.245861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:42.244436 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" event={"ID":"db0c730c-60b9-4401-b27b-3f8459333391","Type":"ContainerStarted","Data":"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0"} Oct 29 22:25:42.245861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:42.245803 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:46.436835 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:46.436796 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:25:49.391510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:49.391479 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:25:49.439218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:49.439168 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" podStartSLOduration=10.050007786 podStartE2EDuration="10.43915468s" totalImagesPullingTime="389.146894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:42.322220307 +0000 UTC m=+2601.170697152" watchObservedRunningTime="2026-10-29 22:25:49.43915468 +0000 UTC m=+2608.287631550" Oct 29 22:25:54.048830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:54.048797 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:25:54.049404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:54.049022 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" containerID="cri-o://42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03" gracePeriod=30 Oct 29 22:25:59.355337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.355302 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:25:59.400245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400211 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400299 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400343 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400381 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400432 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400573 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400614 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.400706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400688 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.400941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400717 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.400941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400759 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.400941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400776 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.400941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.400829 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\") pod \"3c73c94d-3dc2-4c12-8331-434de7e0009d\" (UID: \"3c73c94d-3dc2-4c12-8331-434de7e0009d\") " Oct 29 22:25:59.401152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401008 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:59.401830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401801 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c73c94d-3dc2-4c12-8331-434de7e0009d-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.401830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401833 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.402043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401847 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.402043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401862 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.402043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.401875 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.402663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.402637 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.402881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.402864 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.403026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.402888 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "kube-api-access-jhwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.403277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.403253 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo" pod "3c73c94d-3dc2-4c12-8331-434de7e0009d" (UID: "3c73c94d-3dc2-4c12-8331-434de7e0009d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:25:59.503205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.503169 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhwc8\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-kube-api-access-jhwc8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.503205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.503199 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.503205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.503210 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.503455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.503218 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c73c94d-3dc2-4c12-8331-434de7e0009d-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.866287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.866255 4376 generic.go:356] "Generic (PLEG): container finished" podID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerID="42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03" exitCode=0 Oct 29 22:25:59.866486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.866311 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" Oct 29 22:25:59.866486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.866345 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" event={"ID":"3c73c94d-3dc2-4c12-8331-434de7e0009d","Type":"ContainerDied","Data":"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03"} Oct 29 22:25:59.866486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.866405 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt" event={"ID":"3c73c94d-3dc2-4c12-8331-434de7e0009d","Type":"ContainerDied","Data":"7087d53201e414e414b4e0cddaf43d6893e3a0c35196961fb49b630daf3b46bd"} Oct 29 22:25:59.866486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.866429 4376 scope.go:122] "RemoveContainer" containerID="42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03" Oct 29 22:25:59.879763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.879742 4376 scope.go:122] "RemoveContainer" containerID="42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03" Oct 29 22:25:59.880159 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:25:59.880133 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03\": container with ID starting with 42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03 not found: ID does not exist" containerID="42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03" Oct 29 22:25:59.880240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.880167 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03"} err="failed to get container status \"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03\": rpc error: code = NotFound desc = could not find container \"42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03\": container with ID starting with 42d76eb0e2ec1ead4438b6c82d4f4aa705fd5b6813c9a0fde2b8e6aa90118f03 not found: ID does not exist" Oct 29 22:25:59.947474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.947435 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:25:59.952496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:25:59.952466 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-l6dmt"] Oct 29 22:26:01.276318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:01.276284 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" path="/var/lib/kubelet/pods/3c73c94d-3dc2-4c12-8331-434de7e0009d/volumes" Oct 29 22:26:13.948648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:13.948605 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:26:13.949187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:13.948886 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" podUID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" containerName="maas-api" containerID="cri-o://a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb" gracePeriod=30 Oct 29 22:26:14.593932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.590588 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:26:14.728294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.725577 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\" (UniqueName: \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\") pod \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " Oct 29 22:26:14.728294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.725673 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\") pod \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " Oct 29 22:26:14.745732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.745685 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\") pod \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " Oct 29 22:26:14.746002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.745758 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\") pod \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\" (UID: \"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9\") " Oct 29 22:26:14.753925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.747513 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca" pod "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" (UID: "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:26:14.761055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.760361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs" pod "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" (UID: "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:14.774920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.773179 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls" pod "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" (UID: "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:14.788315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.778971 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944" pod "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" (UID: "a79c2fbf-c1ec-45ca-a5c6-79d9838031e9"). InnerVolumeSpecName "kube-api-access-7c944". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:14.848461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.848428 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c944\" (UniqueName: \"kubernetes.io/projected/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-kube-api-access-7c944\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:14.848461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.848461 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:14.848679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.848472 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:14.848679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.848481 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:14.996512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.996423 4376 generic.go:356] "Generic (PLEG): container finished" podID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" containerID="a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb" exitCode=0 Oct 29 22:26:14.996512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.996469 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" Oct 29 22:26:14.997017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.996512 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" event={"ID":"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9","Type":"ContainerDied","Data":"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb"} Oct 29 22:26:14.997017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.996595 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz" event={"ID":"a79c2fbf-c1ec-45ca-a5c6-79d9838031e9","Type":"ContainerDied","Data":"59b445f0a41e8db5fbeb93412d0a5e3c2f68b8b0e85e7829f5e938c5409698cf"} Oct 29 22:26:14.997017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:14.996622 4376 scope.go:122] "RemoveContainer" containerID="a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb" Oct 29 22:26:15.007613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:15.007591 4376 scope.go:122] "RemoveContainer" containerID="a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb" Oct 29 22:26:15.007871 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:26:15.007848 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb\": container with ID starting with a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb not found: ID does not exist" containerID="a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb" Oct 29 22:26:15.007942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:15.007880 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb"} err="failed to get container status \"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb\": rpc error: code = NotFound desc = could not find container \"a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb\": container with ID starting with a62c28a6602970bee2ac48a9ba2f956d1dae69ac37c5202406e819175ece4eeb not found: ID does not exist" Oct 29 22:26:15.078062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:15.078028 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:26:15.086281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:15.086253 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-529e60-5b5b8b9b48-lf4bz"] Oct 29 22:26:15.311672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:15.311635 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" path="/var/lib/kubelet/pods/a79c2fbf-c1ec-45ca-a5c6-79d9838031e9/volumes" Oct 29 22:26:16.435498 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:26:16.435458 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:26:23.549655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:23.549619 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:26:23.550170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:23.549966 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" containerID="cri-o://8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc" gracePeriod=30 Oct 29 22:26:26.106806 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.106770 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:26:26.107878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107854 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" containerName="payload-processing" Oct 29 22:26:26.107878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107878 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" containerName="payload-processing" Oct 29 22:26:26.108055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107896 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" Oct 29 22:26:26.108055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107916 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" Oct 29 22:26:26.108055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107946 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" containerName="maas-api" Oct 29 22:26:26.108055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.107951 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" containerName="maas-api" Oct 29 22:26:26.108252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.108084 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79c2fbf-c1ec-45ca-a5c6-79d9838031e9" containerName="maas-api" Oct 29 22:26:26.108252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.108100 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97ceb3f-3df7-4f11-827e-ff1392bb00f7" containerName="payload-processing" Oct 29 22:26:26.108252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.108108 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c73c94d-3dc2-4c12-8331-434de7e0009d" containerName="istio-proxy" Oct 29 22:26:26.113829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.113803 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.133811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.133772 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7mr\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.134197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.134169 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.136439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.136413 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.136569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.136452 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.136569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.136493 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.136736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.136705 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.136986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.136963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.137070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.137018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.137134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.137075 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.143106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.143078 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:26:26.146193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.146167 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-749f72-openshift-default-dockercfg-4dm76\"" Oct 29 22:26:26.238864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.238832 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239046 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239099 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239134 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239192 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7mr\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239331 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239333 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239441 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239472 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239512 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239673 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239767 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.239870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.239852 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.241960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.241935 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.242222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.242203 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.264415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.264389 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.264757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.264734 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7mr\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:26.445982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:26.443675 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:27.011609 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:26:27.011570 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48adcf8c_be0d_487e_a79a_a4cf4e066126.slice/crio-840ea6c982bcd67f61f22527b935430a079daca3f02fad210ab8c40b0f4e5c5a WatchSource:0}: Error finding container 840ea6c982bcd67f61f22527b935430a079daca3f02fad210ab8c40b0f4e5c5a: Status 404 returned error can't find the container with id 840ea6c982bcd67f61f22527b935430a079daca3f02fad210ab8c40b0f4e5c5a Oct 29 22:26:27.011755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.011618 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:26:27.014755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.014719 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:26:27.014883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.014787 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:26:27.014883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.014815 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:26:27.151756 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.151703 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerStarted","Data":"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e"} Oct 29 22:26:27.152400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.151791 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerStarted","Data":"840ea6c982bcd67f61f22527b935430a079daca3f02fad210ab8c40b0f4e5c5a"} Oct 29 22:26:27.208849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.208793 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podStartSLOduration=1.20877837 podStartE2EDuration="1.20877837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:27.206117989 +0000 UTC m=+2646.054594848" watchObservedRunningTime="2026-10-29 22:26:27.20877837 +0000 UTC m=+2646.057255214" Oct 29 22:26:27.444707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:27.444614 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:28.451359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.451325 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:28.849620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.849573 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:26:28.855386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855353 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855416 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855483 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855512 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855634 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855735 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855730 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:26:28.855954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855922 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.855954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.855927 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:26:28.856079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.856006 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.856079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.856044 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\") pod \"5523068f-66af-4a90-ad67-035ed3f37fb4\" (UID: \"5523068f-66af-4a90-ad67-035ed3f37fb4\") " Oct 29 22:26:28.856495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.856421 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:26:28.856495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.856475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:26:28.856638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.856501 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:26:28.857067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857044 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.857135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857072 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5523068f-66af-4a90-ad67-035ed3f37fb4-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.857135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857084 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.857135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857098 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.857135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857111 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.857812 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857789 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:28.857959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.857826 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:26:28.858094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.858079 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:26:28.858332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.858307 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4" pod "5523068f-66af-4a90-ad67-035ed3f37fb4" (UID: "5523068f-66af-4a90-ad67-035ed3f37fb4"). InnerVolumeSpecName "kube-api-access-4j2w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:28.959274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.959235 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.959274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.959274 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.959476 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.959287 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5523068f-66af-4a90-ad67-035ed3f37fb4-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:28.959476 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:28.959301 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j2w4\" (UniqueName: \"kubernetes.io/projected/5523068f-66af-4a90-ad67-035ed3f37fb4-kube-api-access-4j2w4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:29.168413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168322 4376 generic.go:356] "Generic (PLEG): container finished" podID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerID="8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc" exitCode=0 Oct 29 22:26:29.168413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168372 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" Oct 29 22:26:29.168413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168399 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerDied","Data":"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc"} Oct 29 22:26:29.168698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168454 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf" event={"ID":"5523068f-66af-4a90-ad67-035ed3f37fb4","Type":"ContainerDied","Data":"5579fa1bc5337956c1608a3c064398f1e62288ad46d88d57691412883bf440d9"} Oct 29 22:26:29.168698 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168476 4376 scope.go:122] "RemoveContainer" containerID="8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc" Oct 29 22:26:29.168813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.168749 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:29.169744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.169731 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:26:29.181396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.181379 4376 scope.go:122] "RemoveContainer" containerID="acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd" Oct 29 22:26:29.191833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.191811 4376 scope.go:122] "RemoveContainer" containerID="8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc" Oct 29 22:26:29.192106 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:26:29.192088 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc\": container with ID starting with 8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc not found: ID does not exist" containerID="8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc" Oct 29 22:26:29.192163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.192117 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc"} err="failed to get container status \"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc\": rpc error: code = NotFound desc = could not find container \"8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc\": container with ID starting with 8fe1dad456a40de975cf785d3a61213c2c3f5a0576bad9313f068ddb4ef411cc not found: ID does not exist" Oct 29 22:26:29.192163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.192136 4376 scope.go:122] "RemoveContainer" containerID="acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd" Oct 29 22:26:29.192384 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:26:29.192364 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd\": container with ID starting with acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd not found: ID does not exist" containerID="acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd" Oct 29 22:26:29.192432 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.192391 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd"} err="failed to get container status \"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd\": rpc error: code = NotFound desc = could not find container \"acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd\": container with ID starting with acf333cbc13fcac66ec32a3addaac68634cff9bd7a3b96c02bc1611c90e437bd not found: ID does not exist" Oct 29 22:26:29.300502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.300466 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:26:29.307003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:29.306969 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-529e60-openshift-default-85bd5958df-pdhvf"] Oct 29 22:26:31.275302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:31.275270 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" path="/var/lib/kubelet/pods/5523068f-66af-4a90-ad67-035ed3f37fb4/volumes" Oct 29 22:26:46.435323 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:26:46.435242 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:26:49.626606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.624985 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:26:49.672165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.672119 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.675158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.674550 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.675158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.674650 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.675158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.674658 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.675158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.674971 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.675158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.675012 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5523068f-66af-4a90-ad67-035ed3f37fb4" containerName="istio-proxy" Oct 29 22:26:49.686644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.686586 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:26:49.687104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.687084 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.718753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.718714 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"e2e-cmp-legacy-30cbe5-provider-cred\"" Oct 29 22:26:49.719005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.718958 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"kube-root-ca.crt\"" Oct 29 22:26:49.719005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.718993 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-50d4af\"" Oct 29 22:26:49.719005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.719020 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"routing-overlay\"" Oct 29 22:26:49.719005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.719026 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"openshift-service-ca.crt\"" Oct 29 22:26:49.719005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.719049 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-e2e--0fbd5c57-dockercfg-csr6r\"" Oct 29 22:26:49.740278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.740247 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pktr\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.740576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.740558 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.740788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.740770 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.740860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.740842 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.873688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.873648 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.874111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.874085 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.874425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.874363 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pktr\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.889829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.875723 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.889829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.876512 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.908928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.903203 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.927927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.921487 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:49.938924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:49.929592 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pktr\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:50.025929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.015669 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:50.402699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.402672 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:26:50.403499 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:26:50.403464 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0dfabc_3859_4fb5_97af_71e0a7e7cc62.slice/crio-4da4d64353aee9eec4cb76d1bd8dd415771c66082275f28e016483fcaa8e0f45 WatchSource:0}: Error finding container 4da4d64353aee9eec4cb76d1bd8dd415771c66082275f28e016483fcaa8e0f45: Status 404 returned error can't find the container with id 4da4d64353aee9eec4cb76d1bd8dd415771c66082275f28e016483fcaa8e0f45 Oct 29 22:26:50.661502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.661393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" event={"ID":"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62","Type":"ContainerStarted","Data":"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2"} Oct 29 22:26:50.661502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.661455 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" event={"ID":"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62","Type":"ContainerStarted","Data":"4da4d64353aee9eec4cb76d1bd8dd415771c66082275f28e016483fcaa8e0f45"} Oct 29 22:26:50.661502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.661494 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:26:50.706445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:50.706386 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" podStartSLOduration=1.706368784 podStartE2EDuration="1.706368784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:50.704658847 +0000 UTC m=+2669.553135690" watchObservedRunningTime="2026-10-29 22:26:50.706368784 +0000 UTC m=+2669.554845628" Oct 29 22:26:51.915157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.915124 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:26:51.920193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.920174 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:51.938381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.938329 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-plugins-e2e-extmod-b-07e166\"" Oct 29 22:26:51.939420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.939395 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:26:51.941844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.941743 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"routing-overlay\"" Oct 29 22:26:51.941844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.941816 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"openshift-service-ca.crt\"" Oct 29 22:26:51.945786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.945755 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"kube-root-ca.crt\"" Oct 29 22:26:51.951025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.951003 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-b9b88c959337\"" Oct 29 22:26:51.951199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.951023 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-58ea3cdf2b14\"" Oct 29 22:26:51.951292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.951084 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-e2e--d445abcd-dockercfg-f5q7z\"" Oct 29 22:26:51.998729 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.998688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:51.999006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.998990 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:51.999075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.999045 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:51.999206 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:51.999174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqxf\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.101611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.101566 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.102073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.102044 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.102261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.102245 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqxf\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.102447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.102431 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.108666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.108640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.109868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.109845 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.131358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.131323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.145326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.145296 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqxf\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.239337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.239295 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:52.761355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:52.761323 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:26:52.763044 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:26:52.763011 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c981a2_95e2_4dbd_be51_00a8e8385e19.slice/crio-a7eda4ef9d1f0112fea62876c14a9b737216277d901e66931296c165a8e008a0 WatchSource:0}: Error finding container a7eda4ef9d1f0112fea62876c14a9b737216277d901e66931296c165a8e008a0: Status 404 returned error can't find the container with id a7eda4ef9d1f0112fea62876c14a9b737216277d901e66931296c165a8e008a0 Oct 29 22:26:53.687195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:53.687154 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" event={"ID":"b2c981a2-95e2-4dbd-be51-00a8e8385e19","Type":"ContainerStarted","Data":"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c"} Oct 29 22:26:53.687616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:53.687221 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" event={"ID":"b2c981a2-95e2-4dbd-be51-00a8e8385e19","Type":"ContainerStarted","Data":"a7eda4ef9d1f0112fea62876c14a9b737216277d901e66931296c165a8e008a0"} Oct 29 22:26:53.687616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:53.687251 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:26:53.736778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:26:53.736724 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" podStartSLOduration=2.7367115010000003 podStartE2EDuration="2.736711501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:53.734886153 +0000 UTC m=+2672.583363000" watchObservedRunningTime="2026-10-29 22:26:53.736711501 +0000 UTC m=+2672.585188354" Oct 29 22:27:01.672784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:01.672752 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:27:03.805384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.805342 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:27:03.807975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.807951 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:27:03.844567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.844532 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:03.851606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.851575 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:03.868216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.868186 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:03.991254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.991204 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:03.991556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.991531 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mmh\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:03.991780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.991761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:03.991932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:03.991897 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.248020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.247978 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.248326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.248305 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mmh\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.248846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.248811 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.249078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.249049 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.262480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.262445 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.274981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.269224 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.294336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.294306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.297458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.297399 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mmh\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.471068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.471024 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:04.787112 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:27:04.787079 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f91037f_5747_4006_8640_523af0c15135.slice/crio-dbde48b3fc6bb44e69d746e4a4ca1f4042fdedacbc465f7b974f86af63c72850 WatchSource:0}: Error finding container dbde48b3fc6bb44e69d746e4a4ca1f4042fdedacbc465f7b974f86af63c72850: Status 404 returned error can't find the container with id dbde48b3fc6bb44e69d746e4a4ca1f4042fdedacbc465f7b974f86af63c72850 Oct 29 22:27:04.787112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.787088 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:04.832727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:04.832696 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" event={"ID":"5f91037f-5747-4006-8640-523af0c15135","Type":"ContainerStarted","Data":"dbde48b3fc6bb44e69d746e4a4ca1f4042fdedacbc465f7b974f86af63c72850"} Oct 29 22:27:05.842399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:05.842362 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" event={"ID":"5f91037f-5747-4006-8640-523af0c15135","Type":"ContainerStarted","Data":"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc"} Oct 29 22:27:05.842895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:05.842503 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:05.888973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:05.888916 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" podStartSLOduration=2.888884902 podStartE2EDuration="2.888884902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:05.887141585 +0000 UTC m=+2684.735618432" watchObservedRunningTime="2026-10-29 22:27:05.888884902 +0000 UTC m=+2684.737361770" Oct 29 22:27:08.805703 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:08.805646 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" podUID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" containerName="payload-processing" containerID="cri-o://195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c" gracePeriod=25 Oct 29 22:27:09.165117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.165086 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:27:09.245849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.245811 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\") pod \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " Oct 29 22:27:09.246063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.245998 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\") pod \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " Oct 29 22:27:09.246153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246073 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\") pod \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " Oct 29 22:27:09.246153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246120 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\") pod \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\" (UID: \"b2c981a2-95e2-4dbd-be51-00a8e8385e19\") " Oct 29 22:27:09.246276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246252 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume" pod "b2c981a2-95e2-4dbd-be51-00a8e8385e19" (UID: "b2c981a2-95e2-4dbd-be51-00a8e8385e19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:09.246435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246413 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume" pod "b2c981a2-95e2-4dbd-be51-00a8e8385e19" (UID: "b2c981a2-95e2-4dbd-be51-00a8e8385e19"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:09.246840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246820 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:09.246840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.246839 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c981a2-95e2-4dbd-be51-00a8e8385e19-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:09.248222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.248197 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf" pod "b2c981a2-95e2-4dbd-be51-00a8e8385e19" (UID: "b2c981a2-95e2-4dbd-be51-00a8e8385e19"). InnerVolumeSpecName "kube-api-access-9sqxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:09.248324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.248255 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials" pod "b2c981a2-95e2-4dbd-be51-00a8e8385e19" (UID: "b2c981a2-95e2-4dbd-be51-00a8e8385e19"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:09.349298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.349270 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:09.349298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.349295 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqxf\" (UniqueName: \"kubernetes.io/projected/b2c981a2-95e2-4dbd-be51-00a8e8385e19-kube-api-access-9sqxf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:09.877468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.877428 4376 generic.go:356] "Generic (PLEG): container finished" podID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" containerID="195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c" exitCode=0 Oct 29 22:27:09.877468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.877468 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" Oct 29 22:27:09.878002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.877492 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" event={"ID":"b2c981a2-95e2-4dbd-be51-00a8e8385e19","Type":"ContainerDied","Data":"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c"} Oct 29 22:27:09.878002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.877550 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g" event={"ID":"b2c981a2-95e2-4dbd-be51-00a8e8385e19","Type":"ContainerDied","Data":"a7eda4ef9d1f0112fea62876c14a9b737216277d901e66931296c165a8e008a0"} Oct 29 22:27:09.878002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.877579 4376 scope.go:122] "RemoveContainer" containerID="195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c" Oct 29 22:27:09.888794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.888772 4376 scope.go:122] "RemoveContainer" containerID="195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c" Oct 29 22:27:09.889117 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:09.889097 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c\": container with ID starting with 195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c not found: ID does not exist" containerID="195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c" Oct 29 22:27:09.889185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.889126 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c"} err="failed to get container status \"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c\": rpc error: code = NotFound desc = could not find container \"195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c\": container with ID starting with 195d9f79072e3154f55a1b86e7b27ab33c5e5454e9c247abb101361814716c1c not found: ID does not exist" Oct 29 22:27:09.934956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.934924 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:27:09.948575 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:09.948537 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-5888xnd9g"] Oct 29 22:27:11.290344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:11.290298 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" path="/var/lib/kubelet/pods/b2c981a2-95e2-4dbd-be51-00a8e8385e19/volumes" Oct 29 22:27:15.146065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:15.146028 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:15.148969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:15.148940 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:16.436055 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:16.436013 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:27:20.146952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.146879 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" podUID="5f91037f-5747-4006-8640-523af0c15135" containerName="payload-processing" containerID="cri-o://536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc" gracePeriod=25 Oct 29 22:27:20.465497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.465469 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:20.500843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.500810 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\") pod \"5f91037f-5747-4006-8640-523af0c15135\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " Oct 29 22:27:20.501085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.500879 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\") pod \"5f91037f-5747-4006-8640-523af0c15135\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " Oct 29 22:27:20.501085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.500990 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\") pod \"5f91037f-5747-4006-8640-523af0c15135\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " Oct 29 22:27:20.501085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.501071 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\") pod \"5f91037f-5747-4006-8640-523af0c15135\" (UID: \"5f91037f-5747-4006-8640-523af0c15135\") " Oct 29 22:27:20.501370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.501324 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume" pod "5f91037f-5747-4006-8640-523af0c15135" (UID: "5f91037f-5747-4006-8640-523af0c15135"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:20.501672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.501647 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume" pod "5f91037f-5747-4006-8640-523af0c15135" (UID: "5f91037f-5747-4006-8640-523af0c15135"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:20.502090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.502068 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.502151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.502098 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f91037f-5747-4006-8640-523af0c15135-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.503281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.503252 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh" pod "5f91037f-5747-4006-8640-523af0c15135" (UID: "5f91037f-5747-4006-8640-523af0c15135"). InnerVolumeSpecName "kube-api-access-n8mmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:20.503374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.503283 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials" pod "5f91037f-5747-4006-8640-523af0c15135" (UID: "5f91037f-5747-4006-8640-523af0c15135"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:20.603659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.603614 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mmh\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-kube-api-access-n8mmh\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.603659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.603648 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f91037f-5747-4006-8640-523af0c15135-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.979666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.979628 4376 generic.go:356] "Generic (PLEG): container finished" podID="5f91037f-5747-4006-8640-523af0c15135" containerID="536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc" exitCode=0 Oct 29 22:27:20.979851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.979685 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" Oct 29 22:27:20.979851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.979717 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" event={"ID":"5f91037f-5747-4006-8640-523af0c15135","Type":"ContainerDied","Data":"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc"} Oct 29 22:27:20.979851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.979777 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv" event={"ID":"5f91037f-5747-4006-8640-523af0c15135","Type":"ContainerDied","Data":"dbde48b3fc6bb44e69d746e4a4ca1f4042fdedacbc465f7b974f86af63c72850"} Oct 29 22:27:20.979851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.979806 4376 scope.go:122] "RemoveContainer" containerID="536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc" Oct 29 22:27:20.991334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.991312 4376 scope.go:122] "RemoveContainer" containerID="536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc" Oct 29 22:27:20.991597 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:20.991577 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc\": container with ID starting with 536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc not found: ID does not exist" containerID="536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc" Oct 29 22:27:20.991652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:20.991606 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc"} err="failed to get container status \"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc\": rpc error: code = NotFound desc = could not find container \"536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc\": container with ID starting with 536687927b99acb34aba992051a2af56c73750b8dc539612f425cb85e62d62dc not found: ID does not exist" Oct 29 22:27:21.051858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:21.051823 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:21.069826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:21.069794 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dxntbv"] Oct 29 22:27:21.275655 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:21.275573 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f91037f-5747-4006-8640-523af0c15135" path="/var/lib/kubelet/pods/5f91037f-5747-4006-8640-523af0c15135/volumes" Oct 29 22:27:30.586560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.586521 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:27:30.587412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587389 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" containerName="payload-processing" Oct 29 22:27:30.587412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587406 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" containerName="payload-processing" Oct 29 22:27:30.587590 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587461 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5f91037f-5747-4006-8640-523af0c15135" containerName="payload-processing" Oct 29 22:27:30.587590 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587466 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f91037f-5747-4006-8640-523af0c15135" containerName="payload-processing" Oct 29 22:27:30.587694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587612 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f91037f-5747-4006-8640-523af0c15135" containerName="payload-processing" Oct 29 22:27:30.587694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.587634 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c981a2-95e2-4dbd-be51-00a8e8385e19" containerName="payload-processing" Oct 29 22:27:30.593834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.593808 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.614007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.613979 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:27:30.618648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.618620 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-749f72-serving-cert\"" Oct 29 22:27:30.697214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.697177 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c7l\" (UniqueName: \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.697400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.697233 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.697469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.697445 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.697513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.697483 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.799144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.799107 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.799144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.799152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.799353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.799326 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c7l\" (UniqueName: \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.799410 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.799360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.800111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.800081 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.801689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.801665 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.801818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.801785 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.857768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.857673 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c7l\" (UniqueName: \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\") pod \"maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:30.915127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:30.915082 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:31.239430 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:27:31.239398 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cbeffd_8427_4f7b_9c9e_8394f6a12a96.slice/crio-3642e1295731e1da0f66ac4aee92e90dfd49a1899cea9a2f19dd28ee3d2f766a WatchSource:0}: Error finding container 3642e1295731e1da0f66ac4aee92e90dfd49a1899cea9a2f19dd28ee3d2f766a: Status 404 returned error can't find the container with id 3642e1295731e1da0f66ac4aee92e90dfd49a1899cea9a2f19dd28ee3d2f766a Oct 29 22:27:31.240160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:31.240132 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:27:32.120532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:32.120491 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" event={"ID":"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96","Type":"ContainerStarted","Data":"5c7dc8665b51f142d5c561f337c398925f6b41957d5de3552d09b385af6a64c2"} Oct 29 22:27:32.121055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:32.120549 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" event={"ID":"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96","Type":"ContainerStarted","Data":"3642e1295731e1da0f66ac4aee92e90dfd49a1899cea9a2f19dd28ee3d2f766a"} Oct 29 22:27:32.121055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:32.120579 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:32.178873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:32.178826 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" podStartSLOduration=1.761809778 podStartE2EDuration="2.178808273s" totalImagesPullingTime="416.998495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:32.175754446 +0000 UTC m=+2711.024231292" watchObservedRunningTime="2026-10-29 22:27:32.178808273 +0000 UTC m=+2711.027285139" Oct 29 22:27:38.192599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:38.192571 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:27:46.488197 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:46.488154 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:27:51.771131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.771091 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:27:51.780303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.780270 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:51.788676 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.788646 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:27:51.809425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809399 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"agc-cred-442becc712bf\"" Oct 29 22:27:51.809577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809485 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-7fe78e\"" Oct 29 22:27:51.809577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809487 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"routing-overlay\"" Oct 29 22:27:51.809577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809544 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"kube-root-ca.crt\"" Oct 29 22:27:51.809577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809553 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"payload-processing-external-model-e2e--eadd026f-dockercfg-xx6km\"" Oct 29 22:27:51.809577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.809575 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7fe78e\"/\"openshift-service-ca.crt\"" Oct 29 22:27:51.902573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.902541 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:51.902733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.902650 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:51.902733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.902677 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b874r\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:51.902822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:51.902780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.032980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.005583 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.035218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.035185 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.049205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.049171 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b874r\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.049525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.049505 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.062921 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.050777 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.065047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.036262 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.065217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.063630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.084172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.084119 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b874r\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\") pod \"payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.115547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.115227 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:52.520533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:52.520503 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:27:52.521595 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:27:52.521568 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f421c5_0f72_4dd3_8fa9_bcc826d4c088.slice/crio-e0f0d332b2b9a732d2b2b00afa273b13ea8095b21033832287baca366712377f WatchSource:0}: Error finding container e0f0d332b2b9a732d2b2b00afa273b13ea8095b21033832287baca366712377f: Status 404 returned error can't find the container with id e0f0d332b2b9a732d2b2b00afa273b13ea8095b21033832287baca366712377f Oct 29 22:27:53.421938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:53.421873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" event={"ID":"43f421c5-0f72-4dd3-8fa9-bcc826d4c088","Type":"ContainerStarted","Data":"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8"} Oct 29 22:27:53.422386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:53.422032 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" event={"ID":"43f421c5-0f72-4dd3-8fa9-bcc826d4c088","Type":"ContainerStarted","Data":"e0f0d332b2b9a732d2b2b00afa273b13ea8095b21033832287baca366712377f"} Oct 29 22:27:53.422386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:53.422148 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:27:53.469551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:53.469499 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" podStartSLOduration=2.469485961 podStartE2EDuration="2.469485961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:53.465166455 +0000 UTC m=+2732.313643335" watchObservedRunningTime="2026-10-29 22:27:53.469485961 +0000 UTC m=+2732.317962858" Oct 29 22:27:54.634782 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:54.634747 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-7fe78e/agc-cred-442becc712bf: secret "agc-cred-442becc712bf" not found Oct 29 22:27:54.634782 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:54.634778 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k: secret "agc-cred-442becc712bf" not found Oct 29 22:27:54.635410 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:54.634843 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials podName:43f421c5-0f72-4dd3-8fa9-bcc826d4c088 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:55.13482405 +0000 UTC m=+2733.983300891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088") : secret "agc-cred-442becc712bf" not found Oct 29 22:27:55.152744 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:55.152703 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-7fe78e/agc-cred-442becc712bf: secret "agc-cred-442becc712bf" not found Oct 29 22:27:55.153146 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:55.153120 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k: secret "agc-cred-442becc712bf" not found Oct 29 22:27:55.153360 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:55.153345 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials podName:43f421c5-0f72-4dd3-8fa9-bcc826d4c088 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:56.153313365 +0000 UTC m=+2735.001790205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088") : secret "agc-cred-442becc712bf" not found Oct 29 22:27:55.381854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:27:55.381820 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:27:56.185947 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:56.184114 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-7fe78e/agc-cred-442becc712bf: secret "agc-cred-442becc712bf" not found Oct 29 22:27:56.185947 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:56.184157 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k: secret "agc-cred-442becc712bf" not found Oct 29 22:27:56.185947 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:56.184235 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials podName:43f421c5-0f72-4dd3-8fa9-bcc826d4c088 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:58.184205627 +0000 UTC m=+2737.032682455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088") : secret "agc-cred-442becc712bf" not found Oct 29 22:27:58.218542 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:58.218504 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-7fe78e/agc-cred-442becc712bf: secret "agc-cred-442becc712bf" not found Oct 29 22:27:58.218542 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:58.218539 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k: secret "agc-cred-442becc712bf" not found Oct 29 22:27:58.219286 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:27:58.218610 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials podName:43f421c5-0f72-4dd3-8fa9-bcc826d4c088 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:02.218590393 +0000 UTC m=+2741.067067236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088") : secret "agc-cred-442becc712bf" not found Oct 29 22:28:00.468743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:00.468702 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" podUID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" containerName="payload-processing" containerID="cri-o://022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8" gracePeriod=25 Oct 29 22:28:01.009601 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.009566 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:28:01.120736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.120697 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\") pod \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " Oct 29 22:28:01.120967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.120943 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\") pod \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " Oct 29 22:28:01.121087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.121069 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\") pod \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " Oct 29 22:28:01.121249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.121232 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\") pod \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\" (UID: \"43f421c5-0f72-4dd3-8fa9-bcc826d4c088\") " Oct 29 22:28:01.121403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.121319 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume" pod "43f421c5-0f72-4dd3-8fa9-bcc826d4c088" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:01.121558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.121532 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume" pod "43f421c5-0f72-4dd3-8fa9-bcc826d4c088" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:01.122016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.121994 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:01.122077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.122022 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:01.123224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.123194 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r" pod "43f421c5-0f72-4dd3-8fa9-bcc826d4c088" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088"). InnerVolumeSpecName "kube-api-access-b874r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:01.123354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.123256 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials" pod "43f421c5-0f72-4dd3-8fa9-bcc826d4c088" (UID: "43f421c5-0f72-4dd3-8fa9-bcc826d4c088"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:01.224016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.223982 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:01.224016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.224012 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b874r\" (UniqueName: \"kubernetes.io/projected/43f421c5-0f72-4dd3-8fa9-bcc826d4c088-kube-api-access-b874r\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:01.518402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.518240 4376 generic.go:356] "Generic (PLEG): container finished" podID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" containerID="022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8" exitCode=0 Oct 29 22:28:01.518402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.518294 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" Oct 29 22:28:01.518402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.518331 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" event={"ID":"43f421c5-0f72-4dd3-8fa9-bcc826d4c088","Type":"ContainerDied","Data":"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8"} Oct 29 22:28:01.518402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.518400 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k" event={"ID":"43f421c5-0f72-4dd3-8fa9-bcc826d4c088","Type":"ContainerDied","Data":"e0f0d332b2b9a732d2b2b00afa273b13ea8095b21033832287baca366712377f"} Oct 29 22:28:01.519141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.518429 4376 scope.go:122] "RemoveContainer" containerID="022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8" Oct 29 22:28:01.529343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.529313 4376 scope.go:122] "RemoveContainer" containerID="022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8" Oct 29 22:28:01.529631 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:28:01.529613 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8\": container with ID starting with 022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8 not found: ID does not exist" containerID="022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8" Oct 29 22:28:01.529687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.529642 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8"} err="failed to get container status \"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8\": rpc error: code = NotFound desc = could not find container \"022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8\": container with ID starting with 022505ddb24bf57f0723097d202182f5fdec9e42d3cae95a36f42429886745f8 not found: ID does not exist" Oct 29 22:28:01.575674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.575635 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:28:01.588472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:01.588436 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7fe78e/payload-processing-external-model-e2e-extmod-a-7fe78e-7b59t2k7k"] Oct 29 22:28:03.275775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:03.275738 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" path="/var/lib/kubelet/pods/43f421c5-0f72-4dd3-8fa9-bcc826d4c088/volumes" Oct 29 22:28:16.435269 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:28:16.435229 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:28:33.508022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.507955 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:28:33.509285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.509264 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" containerName="payload-processing" Oct 29 22:28:33.509285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.509287 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" containerName="payload-processing" Oct 29 22:28:33.509458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.509449 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f421c5-0f72-4dd3-8fa9-bcc826d4c088" containerName="payload-processing" Oct 29 22:28:33.518226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.518200 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.533227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.533197 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:28:33.541174 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.541151 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c06a2c\"" Oct 29 22:28:33.552539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.552518 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c06a2c-dockercfg-59v44\"" Oct 29 22:28:33.642954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.634439 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.642954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.634720 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlbc\" (UniqueName: \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.802982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.802444 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.866938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.857959 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlbc\" (UniqueName: \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.872567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.872527 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:33.923502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:33.923467 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlbc\" (UniqueName: \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\") pod \"payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:34.138827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:34.138740 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:34.451155 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:28:34.451125 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1e3fd0_9ba4_4d71_904a_1ef68df3b059.slice/crio-c38d6265b22e4a1fcd9ca917c2a1f8e794509353ab80c84bbce9c41cceae8024 WatchSource:0}: Error finding container c38d6265b22e4a1fcd9ca917c2a1f8e794509353ab80c84bbce9c41cceae8024: Status 404 returned error can't find the container with id c38d6265b22e4a1fcd9ca917c2a1f8e794509353ab80c84bbce9c41cceae8024 Oct 29 22:28:34.451155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:34.451130 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:28:35.043942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.037566 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:28:35.162456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.162371 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:28:35.162456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.162425 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:35.162735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.162618 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" event={"ID":"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059","Type":"ContainerStarted","Data":"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92"} Oct 29 22:28:35.162735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.162668 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" event={"ID":"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059","Type":"ContainerStarted","Data":"c38d6265b22e4a1fcd9ca917c2a1f8e794509353ab80c84bbce9c41cceae8024"} Oct 29 22:28:35.162869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.162763 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.185671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.185640 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-75ea0c-openshift-default-dockercfg-s7tfk\"" Oct 29 22:28:35.212050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.212003 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.212251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.212223 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfqp\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.212614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.212585 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.212850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.212830 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.213058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.213040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.213200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.213185 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.213669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.213639 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.213782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.213718 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.213782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.213761 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.217029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.216986 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" podStartSLOduration=2.2169747539999998 podStartE2EDuration="2.216974754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:35.168358719 +0000 UTC m=+2774.016835574" watchObservedRunningTime="2026-10-29 22:28:35.216974754 +0000 UTC m=+2774.065451599" Oct 29 22:28:35.321986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.321869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322006 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322046 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322090 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322144 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322234 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322444 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322341 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.322614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.322448 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfqp\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.336935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.334241 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.336935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.335158 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.336935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.335402 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.336935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.335463 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.350987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.345175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.372967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.368493 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.435930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.428523 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.446935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.440626 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfqp\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.462990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.461101 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\") pod \"e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:35.515226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:35.514935 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:36.131118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:36.131090 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:28:36.131964 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:28:36.131938 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b69987_5bf8_4949_82b0_7ca1f4e0eac3.slice/crio-1807aaff8703e9bfadbb08ca49bd87d7fb3572d55d52099d7da4f7499bb4f144 WatchSource:0}: Error finding container 1807aaff8703e9bfadbb08ca49bd87d7fb3572d55d52099d7da4f7499bb4f144: Status 404 returned error can't find the container with id 1807aaff8703e9bfadbb08ca49bd87d7fb3572d55d52099d7da4f7499bb4f144 Oct 29 22:28:36.135532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:36.135495 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:36.135632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:36.135581 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:36.135697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:36.135628 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:37.113956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:37.111840 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerStarted","Data":"d290e41260584899e16f00ca6b124f928cf6bf75a3d29219b7c38b2f52cbf1db"} Oct 29 22:28:37.113956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:37.111929 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerStarted","Data":"1807aaff8703e9bfadbb08ca49bd87d7fb3572d55d52099d7da4f7499bb4f144"} Oct 29 22:28:37.175313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:37.175258 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podStartSLOduration=3.175244265 podStartE2EDuration="3.175244265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:37.171243106 +0000 UTC m=+2776.019719951" watchObservedRunningTime="2026-10-29 22:28:37.175244265 +0000 UTC m=+2776.023721154" Oct 29 22:28:37.515564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:37.515534 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:37.521811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:37.521780 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:38.119594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:38.119562 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:38.120337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:38.120315 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:28:45.211064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:45.211018 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:28:45.211636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:45.211095 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:46.250934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:46.245158 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:28:46.493992 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:28:46.493060 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:28:48.432264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.432232 4376 generic.go:356] "Generic (PLEG): container finished" podID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerID="173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e" exitCode=0 Oct 29 22:28:48.432704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.432300 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerDied","Data":"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e"} Oct 29 22:28:48.432779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.432762 4376 scope.go:122] "RemoveContainer" containerID="173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e" Oct 29 22:28:48.435016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.434972 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:48.435097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.435062 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:48.435133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:48.435095 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:28:49.494931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:49.494872 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerStarted","Data":"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d"} Oct 29 22:28:50.455928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:50.455100 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:28:50.458053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:50.458023 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:28:50.458196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:50.458126 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 29 22:28:51.458930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:51.458288 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:28:51.458930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:51.458357 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 29 22:28:52.452925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:52.446193 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:28:52.452925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:52.446277 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 29 22:28:53.454927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:53.454302 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:28:53.454927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:53.454382 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 29 22:28:54.444310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:54.444276 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 29 22:28:54.444515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:54.444347 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 29 22:28:55.450483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:55.450453 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:28:55.450932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:55.450919 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:28:55.451507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:28:55.451487 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:29:16.435865 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:29:16.435820 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:29:19.926064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:19.926025 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:29:24.927355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:24.927310 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" podUID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" containerName="payload-processing" containerID="cri-o://edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2" gracePeriod=25 Oct 29 22:29:25.250841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.250807 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:29:25.362926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.356148 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\") pod \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " Oct 29 22:29:25.362926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.356218 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\") pod \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " Oct 29 22:29:25.362926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.356667 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\") pod \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " Oct 29 22:29:25.362926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.356719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\") pod \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\" (UID: \"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62\") " Oct 29 22:29:25.364358 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.364070 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume" pod "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" (UID: "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:25.364986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.364961 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume" pod "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" (UID: "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:25.489442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.489409 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:25.502992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.502959 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:25.591414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.591146 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials" pod "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" (UID: "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:25.660942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.635159 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:25.718031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.709082 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr" pod "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" (UID: "2b0dfabc-3859-4fb5-97af-71e0a7e7cc62"). InnerVolumeSpecName "kube-api-access-6pktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:25.818974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:25.818232 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pktr\" (UniqueName: \"kubernetes.io/projected/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62-kube-api-access-6pktr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:26.018967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.018878 4376 generic.go:356] "Generic (PLEG): container finished" podID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" containerID="edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2" exitCode=0 Oct 29 22:29:26.019571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.019217 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" event={"ID":"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62","Type":"ContainerDied","Data":"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2"} Oct 29 22:29:26.019571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.019296 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" event={"ID":"2b0dfabc-3859-4fb5-97af-71e0a7e7cc62","Type":"ContainerDied","Data":"4da4d64353aee9eec4cb76d1bd8dd415771c66082275f28e016483fcaa8e0f45"} Oct 29 22:29:26.019571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.019348 4376 scope.go:122] "RemoveContainer" containerID="edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2" Oct 29 22:29:26.019826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.019751 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj" Oct 29 22:29:26.043864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.043839 4376 scope.go:122] "RemoveContainer" containerID="edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2" Oct 29 22:29:26.044255 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:29:26.044222 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2\": container with ID starting with edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2 not found: ID does not exist" containerID="edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2" Oct 29 22:29:26.044340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.044268 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2"} err="failed to get container status \"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2\": rpc error: code = NotFound desc = could not find container \"edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2\": container with ID starting with edbb3de819e7990bb1c566260b46dbac897d41e69fa745cab402219502b9b7c2 not found: ID does not exist" Oct 29 22:29:26.091916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.091822 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:29:26.103748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.103717 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-5b66kx7jj"] Oct 29 22:29:26.667749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.667709 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:29:26.668801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.668772 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" containerName="payload-processing" Oct 29 22:29:26.668801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.668799 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" containerName="payload-processing" Oct 29 22:29:26.669141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.669124 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" containerName="payload-processing" Oct 29 22:29:26.673146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.673126 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.691356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.691325 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:29:26.698525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.698504 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"e2e-cfl-e08411-provider-cred\"" Oct 29 22:29:26.699319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.699181 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-e2e--0fbd5c57-dockercfg-2nbzj\"" Oct 29 22:29:26.699319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.699295 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-50d4af\"" Oct 29 22:29:26.699319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.699312 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"routing-overlay\"" Oct 29 22:29:26.699559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.699432 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"openshift-service-ca.crt\"" Oct 29 22:29:26.699559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.699497 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"kube-root-ca.crt\"" Oct 29 22:29:26.827930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.826508 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.827930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.826773 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.827930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.826885 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.827930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.827099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxkr\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.929235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.929134 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.929235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.929211 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.929474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.929324 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxkr\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.929474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.929380 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.930073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.930039 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.930198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.930119 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.931631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.931611 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.954437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.954410 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxkr\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:26.988295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:26.988264 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:27.274665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:27.274631 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0dfabc-3859-4fb5-97af-71e0a7e7cc62" path="/var/lib/kubelet/pods/2b0dfabc-3859-4fb5-97af-71e0a7e7cc62/volumes" Oct 29 22:29:27.301385 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:29:27.301352 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fbef2c_d6c9_4163_abd8_134c1b8c5234.slice/crio-f037b48c83996a9c0aa3c0e9ce0e21f3b38a4f34fd736343d026ae3ed33edecd WatchSource:0}: Error finding container f037b48c83996a9c0aa3c0e9ce0e21f3b38a4f34fd736343d026ae3ed33edecd: Status 404 returned error can't find the container with id f037b48c83996a9c0aa3c0e9ce0e21f3b38a4f34fd736343d026ae3ed33edecd Oct 29 22:29:27.301979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:27.301829 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:29:28.039084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:28.039043 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" event={"ID":"09fbef2c-d6c9-4163-abd8-134c1b8c5234","Type":"ContainerStarted","Data":"f3e2d38c3c78b6f1f58000316d2240b0189692a9170739eb875c21a33ddc9b22"} Oct 29 22:29:28.039293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:28.039199 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:28.039293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:28.039230 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" event={"ID":"09fbef2c-d6c9-4163-abd8-134c1b8c5234","Type":"ContainerStarted","Data":"f037b48c83996a9c0aa3c0e9ce0e21f3b38a4f34fd736343d026ae3ed33edecd"} Oct 29 22:29:28.086710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:28.086655 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" podStartSLOduration=2.086641741 podStartE2EDuration="2.086641741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:28.082162898 +0000 UTC m=+2826.930639756" watchObservedRunningTime="2026-10-29 22:29:28.086641741 +0000 UTC m=+2826.935118620" Oct 29 22:29:39.049886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:39.049801 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:29:46.440061 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:29:46.440012 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:29:49.174751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.174707 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:29:49.179470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.179450 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.198513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.198483 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:29:49.207985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.207934 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4366ca\"/\"openshift-service-ca.crt\"" Oct 29 22:29:49.208097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.207994 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4366ca\"/\"kube-root-ca.crt\"" Oct 29 22:29:49.208198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.208157 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4366ca\"/\"default-dockercfg-2cm62\"" Oct 29 22:29:49.208198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.208168 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4366ca\"/\"e2e-usage-sim-238088-cert\"" Oct 29 22:29:49.261950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.261888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwpf\" (UniqueName: \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.262096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.262076 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.363695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.363664 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwpf\" (UniqueName: \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.363888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.363769 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.366484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.366462 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.389058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.389034 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwpf\" (UniqueName: \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\") pod \"e2e-usage-sim-238088-69c4df475b-hlpd2\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.493734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.493701 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:49.804178 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:29:49.804146 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b901937_83eb_40c9_9532_07223880676f.slice/crio-f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8 WatchSource:0}: Error finding container f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8: Status 404 returned error can't find the container with id f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8 Oct 29 22:29:49.804629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:49.804605 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:29:50.282723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:50.282684 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" event={"ID":"4b901937-83eb-40c9-9532-07223880676f","Type":"ContainerStarted","Data":"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a"} Oct 29 22:29:50.283236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:50.282732 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" event={"ID":"4b901937-83eb-40c9-9532-07223880676f","Type":"ContainerStarted","Data":"f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8"} Oct 29 22:29:50.283236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:50.282757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:50.288217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:50.288187 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:29:50.325506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:50.325466 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" podStartSLOduration=1.3254529449999999 podStartE2EDuration="1.325452945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:50.324641522 +0000 UTC m=+2849.173118368" watchObservedRunningTime="2026-10-29 22:29:50.325452945 +0000 UTC m=+2849.173929789" Oct 29 22:29:58.441875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:29:58.441832 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:30:00.183459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.183427 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:00.189818 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.189794 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.202111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.198889 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:00.206105 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.206081 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.206248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.206165 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:00.212469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.212441 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:30:00.243172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.243142 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:00.247404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.247376 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.249417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.249391 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:00.249536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.249437 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:00.249536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.249462 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:00.249899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.249878 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.273805 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.273775 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:00.344511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.344476 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5kr\" (UniqueName: \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.344683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.344596 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvh7\" (UniqueName: \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.344683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.344649 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.344683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.344672 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.344873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.344846 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.345107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.345082 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.345191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.345130 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.345191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.345179 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdltq\" (UniqueName: \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.446605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.446521 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5kr\" (UniqueName: \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.446798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.446623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvh7\" (UniqueName: \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.446798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.446689 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.446798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.446722 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.447029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.446815 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.447029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447007 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.447149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447057 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.447149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447100 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdltq\" (UniqueName: \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.447623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447599 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.447731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447638 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.447731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447645 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.447808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.447793 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.476098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.476071 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5kr\" (UniqueName: \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.476786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.476756 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvh7\" (UniqueName: \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\") pod \"maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.477184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.477163 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.477709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.477690 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdltq\" (UniqueName: \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\") pod \"maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.527678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.527639 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:00.551304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.551278 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:00.574718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.574683 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:00.580359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.580329 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:00.853337 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:00.853306 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c95f36_e537_44ca_a7b9_32e3151790fc.slice/crio-7714398d5525d618b60405e4aa5befbc07d9b79b73b72d2d03311a6db27028dc WatchSource:0}: Error finding container 7714398d5525d618b60405e4aa5befbc07d9b79b73b72d2d03311a6db27028dc: Status 404 returned error can't find the container with id 7714398d5525d618b60405e4aa5befbc07d9b79b73b72d2d03311a6db27028dc Oct 29 22:30:00.853873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.853848 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:00.855173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.855154 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:30:00.894745 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:00.894718 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dea9bbe_c4da_4713_990e_8752c078a52a.slice/crio-665b6c220f0aa6f642a9bd59af095010d2e1a9c1fbb8a0a2963996016d0cc6fb WatchSource:0}: Error finding container 665b6c220f0aa6f642a9bd59af095010d2e1a9c1fbb8a0a2963996016d0cc6fb: Status 404 returned error can't find the container with id 665b6c220f0aa6f642a9bd59af095010d2e1a9c1fbb8a0a2963996016d0cc6fb Oct 29 22:30:00.895148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.895127 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:00.934085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.934057 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:00.939978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:00.939957 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:01.425667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:01.425616 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerStarted","Data":"068046ea9125964b59c2ecb1e72fa42b55729e57423cf6f4965aa94568ae88da"} Oct 29 22:30:01.428069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:01.427764 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerStarted","Data":"e17c872568d188cd5aec6f21da858709e8d1dbe7d4ade5c703670ae9887df4ee"} Oct 29 22:30:01.430348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:01.430272 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerStarted","Data":"665b6c220f0aa6f642a9bd59af095010d2e1a9c1fbb8a0a2963996016d0cc6fb"} Oct 29 22:30:01.449937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:01.436855 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerStarted","Data":"7714398d5525d618b60405e4aa5befbc07d9b79b73b72d2d03311a6db27028dc"} Oct 29 22:30:03.442627 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.442577 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" podUID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" containerName="payload-processing" containerID="cri-o://a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82" gracePeriod=25 Oct 29 22:30:03.826950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.826887 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:30:03.893716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.893680 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:30:03.895608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.894786 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" containerName="payload-processing" Oct 29 22:30:03.895608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.894809 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" containerName="payload-processing" Oct 29 22:30:03.895608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.894969 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" containerName="payload-processing" Oct 29 22:30:03.930127 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.930093 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:30:03.930372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.930187 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:03.944108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.942668 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\") pod \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " Oct 29 22:30:03.944108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.942853 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\") pod \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " Oct 29 22:30:03.944108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.942941 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\") pod \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " Oct 29 22:30:03.944108 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.943013 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\") pod \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\" (UID: \"aa37120d-3fbe-4fe5-b488-247d10a8fb3c\") " Oct 29 22:30:03.946145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.946117 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume" pod "aa37120d-3fbe-4fe5-b488-247d10a8fb3c" (UID: "aa37120d-3fbe-4fe5-b488-247d10a8fb3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:03.946899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.946877 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-31abe10e24c1\"" Oct 29 22:30:03.947042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.946974 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-0caba6281e6e\"" Oct 29 22:30:03.947145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.947101 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-e2e--b7017604-dockercfg-gkpdl\"" Oct 29 22:30:03.947202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.947165 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-2490cdadee51\"" Oct 29 22:30:03.947653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.947604 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume" pod "aa37120d-3fbe-4fe5-b488-247d10a8fb3c" (UID: "aa37120d-3fbe-4fe5-b488-247d10a8fb3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:03.947863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.947843 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"kube-root-ca.crt\"" Oct 29 22:30:03.947997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.947963 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:30:03.948053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.948033 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"openshift-service-ca.crt\"" Oct 29 22:30:03.948172 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.948120 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"routing-overlay\"" Oct 29 22:30:03.956397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.956359 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials" pod "aa37120d-3fbe-4fe5-b488-247d10a8fb3c" (UID: "aa37120d-3fbe-4fe5-b488-247d10a8fb3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:03.958990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:03.958954 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7" pod "aa37120d-3fbe-4fe5-b488-247d10a8fb3c" (UID: "aa37120d-3fbe-4fe5-b488-247d10a8fb3c"). InnerVolumeSpecName "kube-api-access-xfkf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:04.045571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.045480 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.045728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.045633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cd6l\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.045728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.045694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.046036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.046011 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.046373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.046339 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfkf7\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-kube-api-access-xfkf7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.046481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.046377 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.046481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.046394 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.046481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.046407 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa37120d-3fbe-4fe5-b488-247d10a8fb3c-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:04.148807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.148770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.149042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.149003 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.149131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.149069 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cd6l\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.149131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.149120 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.149896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.149868 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.150059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.150000 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.152239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.152213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.174805 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.174772 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cd6l\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.255252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.255218 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:04.474899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.474869 4376 generic.go:356] "Generic (PLEG): container finished" podID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" containerID="a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82" exitCode=0 Oct 29 22:30:04.475287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.474942 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" event={"ID":"aa37120d-3fbe-4fe5-b488-247d10a8fb3c","Type":"ContainerDied","Data":"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82"} Oct 29 22:30:04.475287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.474980 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" Oct 29 22:30:04.475287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.475041 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2" event={"ID":"aa37120d-3fbe-4fe5-b488-247d10a8fb3c","Type":"ContainerDied","Data":"b3464f1d95210b4e61181e16689f9ad501b37749c158f865274d8370f6a480fa"} Oct 29 22:30:04.475287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.475076 4376 scope.go:122] "RemoveContainer" containerID="a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82" Oct 29 22:30:04.488188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.488168 4376 scope.go:122] "RemoveContainer" containerID="a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82" Oct 29 22:30:04.488471 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:04.488448 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82\": container with ID starting with a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82 not found: ID does not exist" containerID="a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82" Oct 29 22:30:04.488533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.488479 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82"} err="failed to get container status \"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82\": rpc error: code = NotFound desc = could not find container \"a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82\": container with ID starting with a094cf11f198c22fed1153d1eec1ab8256289736841e3bea2dfdbe460eb30c82 not found: ID does not exist" Oct 29 22:30:04.532667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.532634 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:30:04.543581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.543550 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749f8hzw2"] Oct 29 22:30:04.578942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:04.578895 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:30:04.640165 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:04.640123 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7986b245_6921_4551_a757_f65743396213.slice/crio-bcf9c40289a99dfae6a0a6f58e165bbe2bd0ade9250c37221e6baf534a23ebe2 WatchSource:0}: Error finding container bcf9c40289a99dfae6a0a6f58e165bbe2bd0ade9250c37221e6baf534a23ebe2: Status 404 returned error can't find the container with id bcf9c40289a99dfae6a0a6f58e165bbe2bd0ade9250c37221e6baf534a23ebe2 Oct 29 22:30:05.275499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.275464 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa37120d-3fbe-4fe5-b488-247d10a8fb3c" path="/var/lib/kubelet/pods/aa37120d-3fbe-4fe5-b488-247d10a8fb3c/volumes" Oct 29 22:30:05.486029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.485996 4376 generic.go:356] "Generic (PLEG): container finished" podID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerID="6723ef20969aabece343ea43ca7b1360536c69d2f71b02d362cd8d2a7309a603" exitCode=60 Oct 29 22:30:05.486615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.486081 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerDied","Data":"6723ef20969aabece343ea43ca7b1360536c69d2f71b02d362cd8d2a7309a603"} Oct 29 22:30:05.486615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.486469 4376 scope.go:122] "RemoveContainer" containerID="6723ef20969aabece343ea43ca7b1360536c69d2f71b02d362cd8d2a7309a603" Oct 29 22:30:05.488276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.488248 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerStarted","Data":"8218b23a297017b3262d0a3e88258ee91954e4bf916e7c171fedd7bb8362b6fe"} Oct 29 22:30:05.490167 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.490142 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerStarted","Data":"af03987b5c9565d5df8b08117d9e304014ce60289ec4c1fa3601895d6b9dd690"} Oct 29 22:30:05.492664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.492627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerStarted","Data":"0183d10b44eff19adfeab2b1e94a6cab8cbf6df2873c58b14ba8f6946a5cdc76"} Oct 29 22:30:05.495229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.495206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" event={"ID":"7986b245-6921-4551-a757-f65743396213","Type":"ContainerStarted","Data":"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa"} Oct 29 22:30:05.495320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.495249 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" event={"ID":"7986b245-6921-4551-a757-f65743396213","Type":"ContainerStarted","Data":"bcf9c40289a99dfae6a0a6f58e165bbe2bd0ade9250c37221e6baf534a23ebe2"} Oct 29 22:30:05.495363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.495320 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:05.571524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.571471 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" podStartSLOduration=2.102517501 podStartE2EDuration="5.571453964s" totalImagesPullingTime="3.468936463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:05.571159853 +0000 UTC m=+2864.419636699" watchObservedRunningTime="2026-10-29 22:30:05.571453964 +0000 UTC m=+2864.419930808" Oct 29 22:30:05.615404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.615353 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" podStartSLOduration=8.61533798 podStartE2EDuration="8.61533798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:05.613797152 +0000 UTC m=+2864.462273997" watchObservedRunningTime="2026-10-29 22:30:05.61533798 +0000 UTC m=+2864.463814848" Oct 29 22:30:05.656300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.656253 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" podStartSLOduration=2.235830674 podStartE2EDuration="5.656238968s" totalImagesPullingTime="3.420408294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:05.6556076 +0000 UTC m=+2864.504084445" watchObservedRunningTime="2026-10-29 22:30:05.656238968 +0000 UTC m=+2864.504715812" Oct 29 22:30:05.700542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:05.700494 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" podStartSLOduration=2.196871206 podStartE2EDuration="5.700481322s" totalImagesPullingTime="3.503610116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:05.696425217 +0000 UTC m=+2864.544902074" watchObservedRunningTime="2026-10-29 22:30:05.700481322 +0000 UTC m=+2864.548958201" Oct 29 22:30:06.506431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.506389 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerID="af03987b5c9565d5df8b08117d9e304014ce60289ec4c1fa3601895d6b9dd690" exitCode=60 Oct 29 22:30:06.506960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.506517 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerDied","Data":"af03987b5c9565d5df8b08117d9e304014ce60289ec4c1fa3601895d6b9dd690"} Oct 29 22:30:06.506960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.506872 4376 scope.go:122] "RemoveContainer" containerID="af03987b5c9565d5df8b08117d9e304014ce60289ec4c1fa3601895d6b9dd690" Oct 29 22:30:06.508686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.508659 4376 generic.go:356] "Generic (PLEG): container finished" podID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerID="0183d10b44eff19adfeab2b1e94a6cab8cbf6df2873c58b14ba8f6946a5cdc76" exitCode=60 Oct 29 22:30:06.508808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.508712 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerDied","Data":"0183d10b44eff19adfeab2b1e94a6cab8cbf6df2873c58b14ba8f6946a5cdc76"} Oct 29 22:30:06.509068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.509049 4376 scope.go:122] "RemoveContainer" containerID="0183d10b44eff19adfeab2b1e94a6cab8cbf6df2873c58b14ba8f6946a5cdc76" Oct 29 22:30:06.511048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.511027 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerStarted","Data":"746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd"} Oct 29 22:30:06.513503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.513470 4376 generic.go:356] "Generic (PLEG): container finished" podID="43f5e148-6141-4cad-a302-521f71f820ce" containerID="8218b23a297017b3262d0a3e88258ee91954e4bf916e7c171fedd7bb8362b6fe" exitCode=60 Oct 29 22:30:06.513627 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.513601 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerDied","Data":"8218b23a297017b3262d0a3e88258ee91954e4bf916e7c171fedd7bb8362b6fe"} Oct 29 22:30:06.513779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.513762 4376 scope.go:122] "RemoveContainer" containerID="8218b23a297017b3262d0a3e88258ee91954e4bf916e7c171fedd7bb8362b6fe" Oct 29 22:30:06.708386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:06.708331 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" podStartSLOduration=3.463090882 podStartE2EDuration="6.708316235s" totalImagesPullingTime="3.245225353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:06.705251036 +0000 UTC m=+2865.553727885" watchObservedRunningTime="2026-10-29 22:30:06.708316235 +0000 UTC m=+2865.556793079" Oct 29 22:30:07.522760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.522725 4376 generic.go:356] "Generic (PLEG): container finished" podID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerID="746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd" exitCode=60 Oct 29 22:30:07.523279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.522802 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerDied","Data":"746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd"} Oct 29 22:30:07.523279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.522858 4376 scope.go:122] "RemoveContainer" containerID="6723ef20969aabece343ea43ca7b1360536c69d2f71b02d362cd8d2a7309a603" Oct 29 22:30:07.523279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.523171 4376 scope.go:122] "RemoveContainer" containerID="746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd" Oct 29 22:30:07.523577 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:07.523552 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2_odh-ai-gateway-infra(c94f30b5-b605-48c3-babd-949f31f6e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" Oct 29 22:30:07.525042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.525017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerStarted","Data":"50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267"} Oct 29 22:30:07.526885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.526866 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerStarted","Data":"09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea"} Oct 29 22:30:07.528949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:07.528887 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerStarted","Data":"c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4"} Oct 29 22:30:08.539093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.539056 4376 generic.go:356] "Generic (PLEG): container finished" podID="43f5e148-6141-4cad-a302-521f71f820ce" containerID="50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267" exitCode=60 Oct 29 22:30:08.539558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.539141 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerDied","Data":"50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267"} Oct 29 22:30:08.539558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.539204 4376 scope.go:122] "RemoveContainer" containerID="8218b23a297017b3262d0a3e88258ee91954e4bf916e7c171fedd7bb8362b6fe" Oct 29 22:30:08.539558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.539423 4376 scope.go:122] "RemoveContainer" containerID="50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267" Oct 29 22:30:08.539815 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:08.539792 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c_odh-ai-gateway-infra(43f5e148-6141-4cad-a302-521f71f820ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" podUID="43f5e148-6141-4cad-a302-521f71f820ce" Oct 29 22:30:08.541295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.541274 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerID="09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea" exitCode=60 Oct 29 22:30:08.541390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.541335 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerDied","Data":"09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea"} Oct 29 22:30:08.541597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.541581 4376 scope.go:122] "RemoveContainer" containerID="09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea" Oct 29 22:30:08.541929 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:08.541889 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd_odh-ai-gateway-infra(8dea9bbe-c4da-4713-990e-8752c078a52a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" Oct 29 22:30:08.543377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.543355 4376 generic.go:356] "Generic (PLEG): container finished" podID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerID="c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4" exitCode=60 Oct 29 22:30:08.543482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.543420 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerDied","Data":"c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4"} Oct 29 22:30:08.543643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.543629 4376 scope.go:122] "RemoveContainer" containerID="c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4" Oct 29 22:30:08.543938 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:08.543880 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4_odh-ai-gateway-infra(49c95f36-e537-44ca-a7b9-32e3151790fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" Oct 29 22:30:08.545385 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.545368 4376 scope.go:122] "RemoveContainer" containerID="746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd" Oct 29 22:30:08.545641 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:08.545621 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2_odh-ai-gateway-infra(c94f30b5-b605-48c3-babd-949f31f6e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" Oct 29 22:30:08.552964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.552946 4376 scope.go:122] "RemoveContainer" containerID="af03987b5c9565d5df8b08117d9e304014ce60289ec4c1fa3601895d6b9dd690" Oct 29 22:30:08.566459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:08.566438 4376 scope.go:122] "RemoveContainer" containerID="0183d10b44eff19adfeab2b1e94a6cab8cbf6df2873c58b14ba8f6946a5cdc76" Oct 29 22:30:09.555001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:09.554967 4376 scope.go:122] "RemoveContainer" containerID="50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267" Oct 29 22:30:09.555461 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:09.555348 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c_odh-ai-gateway-infra(43f5e148-6141-4cad-a302-521f71f820ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" podUID="43f5e148-6141-4cad-a302-521f71f820ce" Oct 29 22:30:09.556667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:09.556645 4376 scope.go:122] "RemoveContainer" containerID="09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea" Oct 29 22:30:09.557025 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:09.556987 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd_odh-ai-gateway-infra(8dea9bbe-c4da-4713-990e-8752c078a52a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" Oct 29 22:30:09.558131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:09.558116 4376 scope.go:122] "RemoveContainer" containerID="c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4" Oct 29 22:30:09.558395 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:09.558377 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4_odh-ai-gateway-infra(49c95f36-e537-44ca-a7b9-32e3151790fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" Oct 29 22:30:16.434568 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:16.434528 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:30:16.516852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:16.516825 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:30:19.275089 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:19.275058 4376 scope.go:122] "RemoveContainer" containerID="09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea" Oct 29 22:30:20.658368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:20.658330 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerID="459874c20d2833b9eecf1b312b0f32590347dd1e8d48794effe476eff4be5dd2" exitCode=60 Oct 29 22:30:20.658781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:20.658398 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerDied","Data":"459874c20d2833b9eecf1b312b0f32590347dd1e8d48794effe476eff4be5dd2"} Oct 29 22:30:20.658781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:20.658455 4376 scope.go:122] "RemoveContainer" containerID="09abc7ba2c23fd6fdaf5bc3b44e09d58b83263770adae23ae34bbc86310f33ea" Oct 29 22:30:20.658781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:20.658750 4376 scope.go:122] "RemoveContainer" containerID="459874c20d2833b9eecf1b312b0f32590347dd1e8d48794effe476eff4be5dd2" Oct 29 22:30:20.659129 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:20.659104 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd_odh-ai-gateway-infra(8dea9bbe-c4da-4713-990e-8752c078a52a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" Oct 29 22:30:21.276269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.276238 4376 scope.go:122] "RemoveContainer" containerID="c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4" Oct 29 22:30:21.276439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.276399 4376 scope.go:122] "RemoveContainer" containerID="746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd" Oct 29 22:30:21.716533 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.716499 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:21.905002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.904975 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:21.966288 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.966252 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:21.967614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967580 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.967614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967603 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.967809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967670 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.967809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967676 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.967809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967796 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.968025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.967813 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:21.972176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.972147 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:21.980159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.978724 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\") pod \"8dea9bbe-c4da-4713-990e-8752c078a52a\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " Oct 29 22:30:21.980159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.978939 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\") pod \"8dea9bbe-c4da-4713-990e-8752c078a52a\" (UID: \"8dea9bbe-c4da-4713-990e-8752c078a52a\") " Oct 29 22:30:21.980536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.980513 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca" pod "8dea9bbe-c4da-4713-990e-8752c078a52a" (UID: "8dea9bbe-c4da-4713-990e-8752c078a52a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:21.982755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.982727 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp" pod "8dea9bbe-c4da-4713-990e-8752c078a52a" (UID: "8dea9bbe-c4da-4713-990e-8752c078a52a"). InnerVolumeSpecName "kube-api-access-9gnqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:21.989499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:21.989472 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:22.082591 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.082531 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.082802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.082762 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbn5\" (UniqueName: \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.082936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.082918 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dea9bbe-c4da-4713-990e-8752c078a52a-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.082995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.082942 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnqp\" (UniqueName: \"kubernetes.io/projected/8dea9bbe-c4da-4713-990e-8752c078a52a-kube-api-access-9gnqp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.186709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.186380 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.186709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.186578 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbn5\" (UniqueName: \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.187333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.187308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.211471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.211443 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbn5\" (UniqueName: \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.264688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.264659 4376 scope.go:122] "RemoveContainer" containerID="50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267" Oct 29 22:30:22.310979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.310895 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:22.678772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.678740 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" Oct 29 22:30:22.678975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.678736 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd" event={"ID":"8dea9bbe-c4da-4713-990e-8752c078a52a","Type":"ContainerDied","Data":"665b6c220f0aa6f642a9bd59af095010d2e1a9c1fbb8a0a2963996016d0cc6fb"} Oct 29 22:30:22.678975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.678899 4376 scope.go:122] "RemoveContainer" containerID="459874c20d2833b9eecf1b312b0f32590347dd1e8d48794effe476eff4be5dd2" Oct 29 22:30:22.681251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.681228 4376 generic.go:356] "Generic (PLEG): container finished" podID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerID="3e82ec6448d98751f51181e375b64a9d79ffc407e57c20b924beff7b4e65924c" exitCode=60 Oct 29 22:30:22.681376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.681305 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerDied","Data":"3e82ec6448d98751f51181e375b64a9d79ffc407e57c20b924beff7b4e65924c"} Oct 29 22:30:22.681811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.681791 4376 scope.go:122] "RemoveContainer" containerID="3e82ec6448d98751f51181e375b64a9d79ffc407e57c20b924beff7b4e65924c" Oct 29 22:30:22.682246 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:22.682217 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4_odh-ai-gateway-infra(49c95f36-e537-44ca-a7b9-32e3151790fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" Oct 29 22:30:22.684376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.684351 4376 generic.go:356] "Generic (PLEG): container finished" podID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerID="3728157ccff4803f764b5f5d77b7b314c45d15abfdef552ff91214ccceefd2bc" exitCode=60 Oct 29 22:30:22.684489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.684434 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerDied","Data":"3728157ccff4803f764b5f5d77b7b314c45d15abfdef552ff91214ccceefd2bc"} Oct 29 22:30:22.684813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.684795 4376 scope.go:122] "RemoveContainer" containerID="3728157ccff4803f764b5f5d77b7b314c45d15abfdef552ff91214ccceefd2bc" Oct 29 22:30:22.685142 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:22.685120 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2_odh-ai-gateway-infra(c94f30b5-b605-48c3-babd-949f31f6e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" Oct 29 22:30:22.692184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.692163 4376 scope.go:122] "RemoveContainer" containerID="c50c21fddc8b00c1757082dae61bf18c51a9b4fa6482af41cac2071941fe1af4" Oct 29 22:30:22.710769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.710747 4376 scope.go:122] "RemoveContainer" containerID="746dd3bd6496930220e0faa862ebb5895578159d88faa6449ad8b012d93ae5cd" Oct 29 22:30:22.718838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.718812 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:22.719183 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:22.718893 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1bfd34_89b1_4041_8eaa_9e22f9e4ded4.slice/crio-fba43d64f91aa5c9be4676e3c6653e611c0d9538b9042bb7cad44a8a10e25d4d WatchSource:0}: Error finding container fba43d64f91aa5c9be4676e3c6653e611c0d9538b9042bb7cad44a8a10e25d4d: Status 404 returned error can't find the container with id fba43d64f91aa5c9be4676e3c6653e611c0d9538b9042bb7cad44a8a10e25d4d Oct 29 22:30:22.739643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.739620 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:22.748361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:22.748266 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888550-q7krd"] Oct 29 22:30:23.274640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.274608 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" path="/var/lib/kubelet/pods/8dea9bbe-c4da-4713-990e-8752c078a52a/volumes" Oct 29 22:30:23.697229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.697149 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerStarted","Data":"4415db477623737f78716c772f906c54ea987a512589cb78da436f9d102f52fe"} Oct 29 22:30:23.697229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.697202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerStarted","Data":"fba43d64f91aa5c9be4676e3c6653e611c0d9538b9042bb7cad44a8a10e25d4d"} Oct 29 22:30:23.700414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.700387 4376 generic.go:356] "Generic (PLEG): container finished" podID="43f5e148-6141-4cad-a302-521f71f820ce" containerID="c3a6e09f396ff9e700d2b5bf6109e3993aeae9557cb492383695342e47d8544f" exitCode=60 Oct 29 22:30:23.700540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.700451 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerDied","Data":"c3a6e09f396ff9e700d2b5bf6109e3993aeae9557cb492383695342e47d8544f"} Oct 29 22:30:23.700540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.700500 4376 scope.go:122] "RemoveContainer" containerID="50cf892699dc5962985715f228e06feeac50c8f12451e4a1db4a3797a44cc267" Oct 29 22:30:23.700846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.700830 4376 scope.go:122] "RemoveContainer" containerID="c3a6e09f396ff9e700d2b5bf6109e3993aeae9557cb492383695342e47d8544f" Oct 29 22:30:23.701204 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:23.701186 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c_odh-ai-gateway-infra(43f5e148-6141-4cad-a302-521f71f820ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" podUID="43f5e148-6141-4cad-a302-521f71f820ce" Oct 29 22:30:23.744930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.744865 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" podStartSLOduration=23.744851358 podStartE2EDuration="23.744851358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:23.739009041 +0000 UTC m=+2882.587485885" watchObservedRunningTime="2026-10-29 22:30:23.744851358 +0000 UTC m=+2882.593328246" Oct 29 22:30:23.795783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.795746 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:23.838448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:23.838415 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:24.004785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.004759 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:24.043235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.043209 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:24.064316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.064283 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:24.065590 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065559 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.065590 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065589 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065643 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065650 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065666 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065673 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065700 4376 cpu_manager.go:414] "containerMap: removing container" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065706 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065718 4376 cpu_manager.go:414] "containerMap: removing container" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065724 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065733 4376 cpu_manager.go:414] "containerMap: removing container" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.065770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065741 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065970 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065990 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.065999 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.066017 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.066034 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dea9bbe-c4da-4713-990e-8752c078a52a" containerName="cleanup" Oct 29 22:30:24.066402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.066043 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.070514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.070490 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.084132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.084017 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:24.113938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.113891 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:24.115341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.115314 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.115341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.115341 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.115580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.115564 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" containerName="cleanup" Oct 29 22:30:24.120368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.120344 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.137800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.137773 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:24.138783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.138756 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\") pod \"c94f30b5-b605-48c3-babd-949f31f6e6e6\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " Oct 29 22:30:24.138895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.138877 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\" (UniqueName: \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\") pod \"49c95f36-e537-44ca-a7b9-32e3151790fc\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " Oct 29 22:30:24.139160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.139037 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\" (UniqueName: \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\") pod \"c94f30b5-b605-48c3-babd-949f31f6e6e6\" (UID: \"c94f30b5-b605-48c3-babd-949f31f6e6e6\") " Oct 29 22:30:24.139245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.139166 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca" pod "c94f30b5-b605-48c3-babd-949f31f6e6e6" (UID: "c94f30b5-b605-48c3-babd-949f31f6e6e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:24.140319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.140294 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\") pod \"49c95f36-e537-44ca-a7b9-32e3151790fc\" (UID: \"49c95f36-e537-44ca-a7b9-32e3151790fc\") " Oct 29 22:30:24.143184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.141000 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca" pod "49c95f36-e537-44ca-a7b9-32e3151790fc" (UID: "49c95f36-e537-44ca-a7b9-32e3151790fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:24.143302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.143098 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq" pod "49c95f36-e537-44ca-a7b9-32e3151790fc" (UID: "49c95f36-e537-44ca-a7b9-32e3151790fc"). InnerVolumeSpecName "kube-api-access-cdltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:24.143368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.143302 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7" pod "c94f30b5-b605-48c3-babd-949f31f6e6e6" (UID: "c94f30b5-b605-48c3-babd-949f31f6e6e6"). InnerVolumeSpecName "kube-api-access-rkvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:24.143570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.143544 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5524\" (UniqueName: \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.143986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.143965 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.144130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.144112 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdltq\" (UniqueName: \"kubernetes.io/projected/49c95f36-e537-44ca-a7b9-32e3151790fc-kube-api-access-cdltq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.144208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.144134 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkvh7\" (UniqueName: \"kubernetes.io/projected/c94f30b5-b605-48c3-babd-949f31f6e6e6-kube-api-access-rkvh7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.144208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.144149 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c95f36-e537-44ca-a7b9-32e3151790fc-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.144208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.144163 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f30b5-b605-48c3-babd-949f31f6e6e6-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:24.245008 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.244887 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.245168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.245035 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5524\" (UniqueName: \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.245168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.245120 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.245293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.245224 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5ch\" (UniqueName: \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.245489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.245467 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.269877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.269850 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5524\" (UniqueName: \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.346915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.346873 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.347037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.347026 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5ch\" (UniqueName: \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.347516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.347496 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.372016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.371987 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5ch\" (UniqueName: \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\") pod \"maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.393196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.393177 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:24.449928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.438407 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:24.712095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.712048 4376 generic.go:356] "Generic (PLEG): container finished" podID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerID="4415db477623737f78716c772f906c54ea987a512589cb78da436f9d102f52fe" exitCode=60 Oct 29 22:30:24.712615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.712563 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerDied","Data":"4415db477623737f78716c772f906c54ea987a512589cb78da436f9d102f52fe"} Oct 29 22:30:24.713653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.713630 4376 scope.go:122] "RemoveContainer" containerID="4415db477623737f78716c772f906c54ea987a512589cb78da436f9d102f52fe" Oct 29 22:30:24.715820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.715795 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" event={"ID":"c94f30b5-b605-48c3-babd-949f31f6e6e6","Type":"ContainerDied","Data":"068046ea9125964b59c2ecb1e72fa42b55729e57423cf6f4965aa94568ae88da"} Oct 29 22:30:24.716078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.716059 4376 scope.go:122] "RemoveContainer" containerID="3728157ccff4803f764b5f5d77b7b314c45d15abfdef552ff91214ccceefd2bc" Oct 29 22:30:24.721987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.721967 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2" Oct 29 22:30:24.727663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.727641 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" Oct 29 22:30:24.727663 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.727650 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4" event={"ID":"49c95f36-e537-44ca-a7b9-32e3151790fc","Type":"ContainerDied","Data":"7714398d5525d618b60405e4aa5befbc07d9b79b73b72d2d03311a6db27028dc"} Oct 29 22:30:24.741367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.741346 4376 scope.go:122] "RemoveContainer" containerID="3e82ec6448d98751f51181e375b64a9d79ffc407e57c20b924beff7b4e65924c" Oct 29 22:30:24.766968 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:24.766945 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1c4d47_4531_49d1_bbd1_739339754669.slice/crio-1be161047a13f00f7533d0d72baf7a7ef26bcb5752f97d9b7a8a95d8ea5665e1 WatchSource:0}: Error finding container 1be161047a13f00f7533d0d72baf7a7ef26bcb5752f97d9b7a8a95d8ea5665e1: Status 404 returned error can't find the container with id 1be161047a13f00f7533d0d72baf7a7ef26bcb5752f97d9b7a8a95d8ea5665e1 Oct 29 22:30:24.767786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.767762 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:24.795887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.795863 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:24.800656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.800632 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:24.809384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.809358 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7fe78e-29888550-qwpc4"] Oct 29 22:30:24.846276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.846243 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:24.856307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.856154 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b7c2d5-29888550-wlct2"] Oct 29 22:30:24.868161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.865733 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:24.940475 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:24.940440 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedada67_d57f_476a_ba7f_6a9204c1f379.slice/crio-d2eaff320851c9e2b22e0b608f55b526e6e1dc3d220124dc86d7dffd89b073b1 WatchSource:0}: Error finding container d2eaff320851c9e2b22e0b608f55b526e6e1dc3d220124dc86d7dffd89b073b1: Status 404 returned error can't find the container with id d2eaff320851c9e2b22e0b608f55b526e6e1dc3d220124dc86d7dffd89b073b1 Oct 29 22:30:24.956945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:24.956920 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:25.018375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.018306 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:25.019728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.019703 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.019835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.019730 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.019835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.019830 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.019977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.019840 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.020171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.020146 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.020258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.020186 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:25.026537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.026515 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.044128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.044081 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:25.081625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.081589 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\") pod \"43f5e148-6141-4cad-a302-521f71f820ce\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " Oct 29 22:30:25.081780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.081747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\" (UniqueName: \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\") pod \"43f5e148-6141-4cad-a302-521f71f820ce\" (UID: \"43f5e148-6141-4cad-a302-521f71f820ce\") " Oct 29 22:30:25.081981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.081953 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca" pod "43f5e148-6141-4cad-a302-521f71f820ce" (UID: "43f5e148-6141-4cad-a302-521f71f820ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:25.082488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.082468 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f5e148-6141-4cad-a302-521f71f820ce-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:25.134545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.134517 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr" pod "43f5e148-6141-4cad-a302-521f71f820ce" (UID: "43f5e148-6141-4cad-a302-521f71f820ce"). InnerVolumeSpecName "kube-api-access-bl5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:25.183929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.183888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.184081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.184069 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq4j\" (UniqueName: \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.184314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.184302 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5kr\" (UniqueName: \"kubernetes.io/projected/43f5e148-6141-4cad-a302-521f71f820ce-kube-api-access-bl5kr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:25.275620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.275560 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c95f36-e537-44ca-a7b9-32e3151790fc" path="/var/lib/kubelet/pods/49c95f36-e537-44ca-a7b9-32e3151790fc/volumes" Oct 29 22:30:25.276197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.276180 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94f30b5-b605-48c3-babd-949f31f6e6e6" path="/var/lib/kubelet/pods/c94f30b5-b605-48c3-babd-949f31f6e6e6/volumes" Oct 29 22:30:25.285361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.285337 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.285480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.285463 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq4j\" (UniqueName: \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.286102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.286080 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.310455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.310428 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq4j\" (UniqueName: \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.357880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.357855 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:25.674652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.674619 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:25.739403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.739369 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerStarted","Data":"bdcdd9c828b8fb15a1c9576ee96ba5bb74c9e37809f69f913f4600b8b79e2874"} Oct 29 22:30:25.739556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.739434 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerStarted","Data":"d2eaff320851c9e2b22e0b608f55b526e6e1dc3d220124dc86d7dffd89b073b1"} Oct 29 22:30:25.741526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.741505 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" Oct 29 22:30:25.741649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.741527 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c" event={"ID":"43f5e148-6141-4cad-a302-521f71f820ce","Type":"ContainerDied","Data":"e17c872568d188cd5aec6f21da858709e8d1dbe7d4ade5c703670ae9887df4ee"} Oct 29 22:30:25.741649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.741588 4376 scope.go:122] "RemoveContainer" containerID="c3a6e09f396ff9e700d2b5bf6109e3993aeae9557cb492383695342e47d8544f" Oct 29 22:30:25.743159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.743133 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerStarted","Data":"80292dcf524134eeb208d99ed56cd09234fbbca42f9674713d8605dabc076fab"} Oct 29 22:30:25.745860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.745374 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerStarted","Data":"9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959"} Oct 29 22:30:25.747178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.747157 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerStarted","Data":"b827f05cf93ad59a1a15a38732ebab88f3705bd126d16c5beb8f93988ea9f231"} Oct 29 22:30:25.747276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.747192 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerStarted","Data":"1be161047a13f00f7533d0d72baf7a7ef26bcb5752f97d9b7a8a95d8ea5665e1"} Oct 29 22:30:25.783654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.783603 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" podStartSLOduration=25.783589433 podStartE2EDuration="25.783589433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.781479865 +0000 UTC m=+2884.629956712" watchObservedRunningTime="2026-10-29 22:30:25.783589433 +0000 UTC m=+2884.632066277" Oct 29 22:30:25.828446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.828384 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" podStartSLOduration=25.828368895 podStartE2EDuration="25.828368895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.823092686 +0000 UTC m=+2884.671569530" watchObservedRunningTime="2026-10-29 22:30:25.828368895 +0000 UTC m=+2884.676845779" Oct 29 22:30:25.909916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.909874 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:25.914416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:25.914391 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e7b56d-29888550-s6m8c"] Oct 29 22:30:26.757155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.757121 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5fed219-8197-459a-827f-7e6e63438780" containerID="64ee33159054e83ab82079facd93cdd8dd1074cbccb93f4fd83b86e677636f19" exitCode=7 Oct 29 22:30:26.757339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.757168 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerDied","Data":"64ee33159054e83ab82079facd93cdd8dd1074cbccb93f4fd83b86e677636f19"} Oct 29 22:30:26.757542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.757525 4376 scope.go:122] "RemoveContainer" containerID="64ee33159054e83ab82079facd93cdd8dd1074cbccb93f4fd83b86e677636f19" Oct 29 22:30:26.759314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.759289 4376 generic.go:356] "Generic (PLEG): container finished" podID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerID="9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959" exitCode=60 Oct 29 22:30:26.759455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.759388 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerDied","Data":"9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959"} Oct 29 22:30:26.759455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.759445 4376 scope.go:122] "RemoveContainer" containerID="4415db477623737f78716c772f906c54ea987a512589cb78da436f9d102f52fe" Oct 29 22:30:26.759647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.759630 4376 scope.go:122] "RemoveContainer" containerID="9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959" Oct 29 22:30:26.759985 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:26.759960 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj_odh-ai-gateway-infra(1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" Oct 29 22:30:26.761295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.761275 4376 generic.go:356] "Generic (PLEG): container finished" podID="ec1c4d47-4531-49d1-bbd1-739339754669" containerID="b827f05cf93ad59a1a15a38732ebab88f3705bd126d16c5beb8f93988ea9f231" exitCode=60 Oct 29 22:30:26.761391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.761346 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerDied","Data":"b827f05cf93ad59a1a15a38732ebab88f3705bd126d16c5beb8f93988ea9f231"} Oct 29 22:30:26.761628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.761612 4376 scope.go:122] "RemoveContainer" containerID="b827f05cf93ad59a1a15a38732ebab88f3705bd126d16c5beb8f93988ea9f231" Oct 29 22:30:26.763230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.763213 4376 generic.go:356] "Generic (PLEG): container finished" podID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerID="bdcdd9c828b8fb15a1c9576ee96ba5bb74c9e37809f69f913f4600b8b79e2874" exitCode=60 Oct 29 22:30:26.763296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.763282 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerDied","Data":"bdcdd9c828b8fb15a1c9576ee96ba5bb74c9e37809f69f913f4600b8b79e2874"} Oct 29 22:30:26.763547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:26.763530 4376 scope.go:122] "RemoveContainer" containerID="bdcdd9c828b8fb15a1c9576ee96ba5bb74c9e37809f69f913f4600b8b79e2874" Oct 29 22:30:27.275980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.275948 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f5e148-6141-4cad-a302-521f71f820ce" path="/var/lib/kubelet/pods/43f5e148-6141-4cad-a302-521f71f820ce/volumes" Oct 29 22:30:27.774028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.773991 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerStarted","Data":"2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1"} Oct 29 22:30:27.776191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.776161 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerStarted","Data":"a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290"} Oct 29 22:30:27.778077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.778054 4376 scope.go:122] "RemoveContainer" containerID="9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959" Oct 29 22:30:27.778424 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:27.778400 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj_odh-ai-gateway-infra(1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" Oct 29 22:30:27.779727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.779707 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerStarted","Data":"a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70"} Oct 29 22:30:27.960639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:27.960592 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" podStartSLOduration=27.960579129 podStartE2EDuration="27.960579129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:27.954755906 +0000 UTC m=+2886.803232751" watchObservedRunningTime="2026-10-29 22:30:27.960579129 +0000 UTC m=+2886.809055974" Oct 29 22:30:28.792686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.792650 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5fed219-8197-459a-827f-7e6e63438780" containerID="a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290" exitCode=7 Oct 29 22:30:28.793136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.792726 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerDied","Data":"a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290"} Oct 29 22:30:28.793136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.792789 4376 scope.go:122] "RemoveContainer" containerID="64ee33159054e83ab82079facd93cdd8dd1074cbccb93f4fd83b86e677636f19" Oct 29 22:30:28.793136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.793112 4376 scope.go:122] "RemoveContainer" containerID="a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290" Oct 29 22:30:28.793543 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:28.793516 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t_odh-ai-gateway-infra(d5fed219-8197-459a-827f-7e6e63438780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" podUID="d5fed219-8197-459a-827f-7e6e63438780" Oct 29 22:30:28.794945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.794899 4376 generic.go:356] "Generic (PLEG): container finished" podID="ec1c4d47-4531-49d1-bbd1-739339754669" containerID="a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70" exitCode=60 Oct 29 22:30:28.795069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.794999 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerDied","Data":"a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70"} Oct 29 22:30:28.795243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.795216 4376 scope.go:122] "RemoveContainer" containerID="a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70" Oct 29 22:30:28.795569 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:28.795548 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2_odh-ai-gateway-infra(ec1c4d47-4531-49d1-bbd1-739339754669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" Oct 29 22:30:28.797176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.797149 4376 generic.go:356] "Generic (PLEG): container finished" podID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerID="2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1" exitCode=60 Oct 29 22:30:28.797294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.797223 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerDied","Data":"2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1"} Oct 29 22:30:28.797453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.797434 4376 scope.go:122] "RemoveContainer" containerID="2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1" Oct 29 22:30:28.797802 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:28.797776 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8_odh-ai-gateway-infra(aedada67-d57f-476a-ba7f-6a9204c1f379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" Oct 29 22:30:28.806517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.806496 4376 scope.go:122] "RemoveContainer" containerID="b827f05cf93ad59a1a15a38732ebab88f3705bd126d16c5beb8f93988ea9f231" Oct 29 22:30:28.820334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:28.820314 4376 scope.go:122] "RemoveContainer" containerID="bdcdd9c828b8fb15a1c9576ee96ba5bb74c9e37809f69f913f4600b8b79e2874" Oct 29 22:30:29.807189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:29.807155 4376 scope.go:122] "RemoveContainer" containerID="a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70" Oct 29 22:30:29.807664 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:29.807495 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2_odh-ai-gateway-infra(ec1c4d47-4531-49d1-bbd1-739339754669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" Oct 29 22:30:29.808892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:29.808875 4376 scope.go:122] "RemoveContainer" containerID="2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1" Oct 29 22:30:29.809199 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:29.809171 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8_odh-ai-gateway-infra(aedada67-d57f-476a-ba7f-6a9204c1f379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" Oct 29 22:30:29.810619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:29.810601 4376 scope.go:122] "RemoveContainer" containerID="a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290" Oct 29 22:30:29.810819 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:29.810802 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t_odh-ai-gateway-infra(d5fed219-8197-459a-827f-7e6e63438780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" podUID="d5fed219-8197-459a-827f-7e6e63438780" Oct 29 22:30:30.571278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:30.571243 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:30:30.571509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:30.571480 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" containerID="cri-o://720b533e7b115f80198d89f98349784c543552b46457b7919b1edc5d43e66913" gracePeriod=30 Oct 29 22:30:35.972360 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:35.960074 4376 generic.go:356] "Generic (PLEG): container finished" podID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerID="720b533e7b115f80198d89f98349784c543552b46457b7919b1edc5d43e66913" exitCode=0 Oct 29 22:30:35.972360 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:35.960381 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" event={"ID":"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93","Type":"ContainerDied","Data":"720b533e7b115f80198d89f98349784c543552b46457b7919b1edc5d43e66913"} Oct 29 22:30:36.577545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.577512 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:30:36.676223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.676189 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:30:36.678088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678057 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" Oct 29 22:30:36.678088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678088 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" Oct 29 22:30:36.678249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678147 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:36.678249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678157 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:36.678535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678373 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" containerName="istio-proxy" Oct 29 22:30:36.678535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.678412 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f5e148-6141-4cad-a302-521f71f820ce" containerName="cleanup" Oct 29 22:30:36.683230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.683199 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:30:36.683314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.683260 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.697126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697103 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697161 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697284 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697459 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697508 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:36.697591 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697544 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697626 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697664 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697742 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.697853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697830 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697895 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697916 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.697846 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\") pod \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\" (UID: \"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93\") " Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.698797 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.698818 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.698832 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.698845 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.698966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.698936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:36.699897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.699875 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-201a3d-serving-cert\"" Oct 29 22:30:36.700139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.700114 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "kube-api-access-l69nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:36.701306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.701281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:36.701391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.701281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:36.702438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.702417 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy" pod "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" (UID: "0f93c982-c57f-4d1d-94b8-31c9a7ee7b93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:36.799952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.799853 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.800109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800041 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h86\" (UniqueName: \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.800258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800238 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.800322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.800434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800423 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.800478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800436 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.800478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800446 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69nq\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-kube-api-access-l69nq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.800478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800454 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.800478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.800462 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:36.902463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.902424 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h86\" (UniqueName: \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.902677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.902657 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.902764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.902748 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.902869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.902853 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.903516 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.903487 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.905404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.905377 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.905531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.905489 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.932341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.932307 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h86\" (UniqueName: \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\") pod \"maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:36.985955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.981303 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" Oct 29 22:30:36.985955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.981309 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm" event={"ID":"0f93c982-c57f-4d1d-94b8-31c9a7ee7b93","Type":"ContainerDied","Data":"e6b3b0d8719ba07ee8861a481408e6855316aceeab33144df13a343a349bd090"} Oct 29 22:30:36.985955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.981447 4376 scope.go:122] "RemoveContainer" containerID="720b533e7b115f80198d89f98349784c543552b46457b7919b1edc5d43e66913" Oct 29 22:30:36.999264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:36.999239 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:37.092187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.085008 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:30:37.101749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.094389 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-vrpkm"] Oct 29 22:30:37.274562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.274521 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f93c982-c57f-4d1d-94b8-31c9a7ee7b93" path="/var/lib/kubelet/pods/0f93c982-c57f-4d1d-94b8-31c9a7ee7b93/volumes" Oct 29 22:30:37.365469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.365437 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:30:37.365606 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:37.365441 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f864ce_c860_4d20_b75f_35b4bd13cee7.slice/crio-1cdc06c10c452962495f7aea1acf4516e60b9aa5a76c37a2e15a718c5563b4b2 WatchSource:0}: Error finding container 1cdc06c10c452962495f7aea1acf4516e60b9aa5a76c37a2e15a718c5563b4b2: Status 404 returned error can't find the container with id 1cdc06c10c452962495f7aea1acf4516e60b9aa5a76c37a2e15a718c5563b4b2 Oct 29 22:30:37.992474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.992435 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" event={"ID":"72f864ce-c860-4d20-b75f-35b4bd13cee7","Type":"ContainerStarted","Data":"3692ba22ecfe8a9726612ebf9d6d6c84243fbe6022d980b0b51cda3a8e6c2503"} Oct 29 22:30:37.992884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.992530 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:37.992884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:37.992550 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" event={"ID":"72f864ce-c860-4d20-b75f-35b4bd13cee7","Type":"ContainerStarted","Data":"1cdc06c10c452962495f7aea1acf4516e60b9aa5a76c37a2e15a718c5563b4b2"} Oct 29 22:30:38.061254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:38.061129 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" podStartSLOduration=47.654908967 podStartE2EDuration="48.061111211s" totalImagesPullingTime="406.202244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:38.053491264 +0000 UTC m=+2896.901968121" watchObservedRunningTime="2026-10-29 22:30:38.061111211 +0000 UTC m=+2896.909588062" Oct 29 22:30:40.265435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:40.265392 4376 scope.go:122] "RemoveContainer" containerID="9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959" Oct 29 22:30:40.265834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:40.265612 4376 scope.go:122] "RemoveContainer" containerID="a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290" Oct 29 22:30:40.266007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:40.265984 4376 scope.go:122] "RemoveContainer" containerID="2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1" Oct 29 22:30:40.371411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:40.370560 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:30:40.371411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:40.370788 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" containerID="cri-o://9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3" gracePeriod=30 Oct 29 22:30:41.024386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:41.024348 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerStarted","Data":"8249e0803a6560a95ed9548eb025fbcc18390f39968a0a22550da771fb85487c"} Oct 29 22:30:41.026353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:41.026327 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerStarted","Data":"661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128"} Oct 29 22:30:41.600344 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:41.600304 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedada67_d57f_476a_ba7f_6a9204c1f379.slice/crio-conmon-661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedada67_d57f_476a_ba7f_6a9204c1f379.slice/crio-661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fed219_8197_459a_827f_7e6e63438780.slice/crio-conmon-28eb21e45d3769e15fd73ec024844dcad76b70ee1bbfe3c1964df9d95295ac69.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:30:42.044561 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.044522 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5fed219-8197-459a-827f-7e6e63438780" containerID="28eb21e45d3769e15fd73ec024844dcad76b70ee1bbfe3c1964df9d95295ac69" exitCode=7 Oct 29 22:30:42.044777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.044643 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerDied","Data":"28eb21e45d3769e15fd73ec024844dcad76b70ee1bbfe3c1964df9d95295ac69"} Oct 29 22:30:42.044777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.044730 4376 scope.go:122] "RemoveContainer" containerID="a8011ee1ca4efc411e67a6015a135bcaad6d1b56a405c53074be03283e8f1290" Oct 29 22:30:42.045398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.045264 4376 scope.go:122] "RemoveContainer" containerID="28eb21e45d3769e15fd73ec024844dcad76b70ee1bbfe3c1964df9d95295ac69" Oct 29 22:30:42.045728 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:42.045701 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t_odh-ai-gateway-infra(d5fed219-8197-459a-827f-7e6e63438780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" podUID="d5fed219-8197-459a-827f-7e6e63438780" Oct 29 22:30:42.047759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.047738 4376 generic.go:356] "Generic (PLEG): container finished" podID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerID="8249e0803a6560a95ed9548eb025fbcc18390f39968a0a22550da771fb85487c" exitCode=60 Oct 29 22:30:42.047966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.047935 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerDied","Data":"8249e0803a6560a95ed9548eb025fbcc18390f39968a0a22550da771fb85487c"} Oct 29 22:30:42.048390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.048370 4376 scope.go:122] "RemoveContainer" containerID="8249e0803a6560a95ed9548eb025fbcc18390f39968a0a22550da771fb85487c" Oct 29 22:30:42.048775 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:42.048747 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj_odh-ai-gateway-infra(1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" Oct 29 22:30:42.050143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.050120 4376 generic.go:356] "Generic (PLEG): container finished" podID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerID="661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128" exitCode=60 Oct 29 22:30:42.050233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.050191 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerDied","Data":"661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128"} Oct 29 22:30:42.050558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.050543 4376 scope.go:122] "RemoveContainer" containerID="661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128" Oct 29 22:30:42.050843 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:42.050823 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8_odh-ai-gateway-infra(aedada67-d57f-476a-ba7f-6a9204c1f379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" Oct 29 22:30:42.068854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.068825 4376 scope.go:122] "RemoveContainer" containerID="9a2f21d5539655ac6b71c70d50055981e1e6b5999511737165d3b2965eda1959" Oct 29 22:30:42.084332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.084305 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:42.092588 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.092539 4376 scope.go:122] "RemoveContainer" containerID="2ac19ce2c5f8878aaad571a26677133b838617a3ebf2d3889c859816c2be68a1" Oct 29 22:30:42.124716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.124685 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:42.285239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:42.276310 4376 scope.go:122] "RemoveContainer" containerID="a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70" Oct 29 22:30:43.063795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.063761 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerStarted","Data":"cd864efddc677ed26bec029fc60442842532ccf7a3f1e55f9b79b6f2597f4419"} Oct 29 22:30:43.106603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.106552 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:43.385133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.385104 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:43.391857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.391829 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:43.415188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.415165 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:43.560587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\" (UniqueName: \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\") pod \"d5fed219-8197-459a-827f-7e6e63438780\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " Oct 29 22:30:43.560787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560609 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\") pod \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " Oct 29 22:30:43.560787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560674 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\" (UniqueName: \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\") pod \"aedada67-d57f-476a-ba7f-6a9204c1f379\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " Oct 29 22:30:43.560977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560835 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\" (UniqueName: \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\") pod \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\" (UID: \"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4\") " Oct 29 22:30:43.560977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560864 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\") pod \"d5fed219-8197-459a-827f-7e6e63438780\" (UID: \"d5fed219-8197-459a-827f-7e6e63438780\") " Oct 29 22:30:43.560977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.560935 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\") pod \"aedada67-d57f-476a-ba7f-6a9204c1f379\" (UID: \"aedada67-d57f-476a-ba7f-6a9204c1f379\") " Oct 29 22:30:43.561813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.561670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca" pod "1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" (UID: "1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:43.561813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.561698 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca" pod "d5fed219-8197-459a-827f-7e6e63438780" (UID: "d5fed219-8197-459a-827f-7e6e63438780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:43.562030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.561955 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca" pod "aedada67-d57f-476a-ba7f-6a9204c1f379" (UID: "aedada67-d57f-476a-ba7f-6a9204c1f379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:43.562572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.562526 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:43.562572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.562569 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fed219-8197-459a-827f-7e6e63438780-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:43.562726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.562586 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedada67-d57f-476a-ba7f-6a9204c1f379-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:43.565386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.565359 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j" pod "d5fed219-8197-459a-827f-7e6e63438780" (UID: "d5fed219-8197-459a-827f-7e6e63438780"). InnerVolumeSpecName "kube-api-access-mcq4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:43.565795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.565770 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5" pod "1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" (UID: "1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4"). InnerVolumeSpecName "kube-api-access-sqbn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:43.566094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.566071 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch" pod "aedada67-d57f-476a-ba7f-6a9204c1f379" (UID: "aedada67-d57f-476a-ba7f-6a9204c1f379"). InnerVolumeSpecName "kube-api-access-dm5ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:43.664784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.664699 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcq4j\" (UniqueName: \"kubernetes.io/projected/d5fed219-8197-459a-827f-7e6e63438780-kube-api-access-mcq4j\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:43.664784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.664731 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5ch\" (UniqueName: \"kubernetes.io/projected/aedada67-d57f-476a-ba7f-6a9204c1f379-kube-api-access-dm5ch\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:43.664784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:43.664740 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbn5\" (UniqueName: \"kubernetes.io/projected/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4-kube-api-access-sqbn5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.007302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.007270 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:30:44.079402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.079367 4376 generic.go:356] "Generic (PLEG): container finished" podID="ec1c4d47-4531-49d1-bbd1-739339754669" containerID="cd864efddc677ed26bec029fc60442842532ccf7a3f1e55f9b79b6f2597f4419" exitCode=60 Oct 29 22:30:44.080055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.079435 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerDied","Data":"cd864efddc677ed26bec029fc60442842532ccf7a3f1e55f9b79b6f2597f4419"} Oct 29 22:30:44.080055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.079500 4376 scope.go:122] "RemoveContainer" containerID="a9df6a47aa62136729231fbf8d08d45e1d01a7c145169b61d775ffdaa5c52f70" Oct 29 22:30:44.080055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.079967 4376 scope.go:122] "RemoveContainer" containerID="cd864efddc677ed26bec029fc60442842532ccf7a3f1e55f9b79b6f2597f4419" Oct 29 22:30:44.080401 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:44.080300 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2_odh-ai-gateway-infra(ec1c4d47-4531-49d1-bbd1-739339754669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" Oct 29 22:30:44.081619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.081598 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" event={"ID":"aedada67-d57f-476a-ba7f-6a9204c1f379","Type":"ContainerDied","Data":"d2eaff320851c9e2b22e0b608f55b526e6e1dc3d220124dc86d7dffd89b073b1"} Oct 29 22:30:44.081728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.081671 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8" Oct 29 22:30:44.083678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.083379 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" event={"ID":"d5fed219-8197-459a-827f-7e6e63438780","Type":"ContainerDied","Data":"80292dcf524134eeb208d99ed56cd09234fbbca42f9674713d8605dabc076fab"} Oct 29 22:30:44.083678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.083385 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t" Oct 29 22:30:44.094152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.085596 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" event={"ID":"1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4","Type":"ContainerDied","Data":"fba43d64f91aa5c9be4676e3c6653e611c0d9538b9042bb7cad44a8a10e25d4d"} Oct 29 22:30:44.094152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.085683 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj" Oct 29 22:30:44.110868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.110631 4376 scope.go:122] "RemoveContainer" containerID="661b1d3185afa477344997c3fbc8921516fb6f95dbd4d75200fe30b5f2750128" Oct 29 22:30:44.125236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.125212 4376 scope.go:122] "RemoveContainer" containerID="28eb21e45d3769e15fd73ec024844dcad76b70ee1bbfe3c1964df9d95295ac69" Oct 29 22:30:44.138535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.138502 4376 scope.go:122] "RemoveContainer" containerID="8249e0803a6560a95ed9548eb025fbcc18390f39968a0a22550da771fb85487c" Oct 29 22:30:44.165912 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.165877 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:44.171469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.171441 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:44.175543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.175517 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0a3d53-29888550-nc4vj"] Oct 29 22:30:44.277010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.265262 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:44.277433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.277411 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-75ea0c-29888550-9qqp8"] Oct 29 22:30:44.284964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.284888 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:44.292096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:44.292069 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888550-xr59t"] Oct 29 22:30:45.217995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.212842 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:30:45.217995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.212965 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:30:45.297538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.297501 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" path="/var/lib/kubelet/pods/1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4/volumes" Oct 29 22:30:45.298155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.298136 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" path="/var/lib/kubelet/pods/aedada67-d57f-476a-ba7f-6a9204c1f379/volumes" Oct 29 22:30:45.298708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.298691 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fed219-8197-459a-827f-7e6e63438780" path="/var/lib/kubelet/pods/d5fed219-8197-459a-827f-7e6e63438780/volumes" Oct 29 22:30:45.329883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.329852 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:45.413938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.413446 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\" (UniqueName: \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\") pod \"ec1c4d47-4531-49d1-bbd1-739339754669\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " Oct 29 22:30:45.413938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.413618 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\") pod \"ec1c4d47-4531-49d1-bbd1-739339754669\" (UID: \"ec1c4d47-4531-49d1-bbd1-739339754669\") " Oct 29 22:30:45.414466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.414434 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca" pod "ec1c4d47-4531-49d1-bbd1-739339754669" (UID: "ec1c4d47-4531-49d1-bbd1-739339754669"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:45.416435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.416414 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524" pod "ec1c4d47-4531-49d1-bbd1-739339754669" (UID: "ec1c4d47-4531-49d1-bbd1-739339754669"). InnerVolumeSpecName "kube-api-access-h5524". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:45.421421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.421397 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5524\" (UniqueName: \"kubernetes.io/projected/ec1c4d47-4531-49d1-bbd1-739339754669-kube-api-access-h5524\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.421521 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.421423 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1c4d47-4531-49d1-bbd1-739339754669-maas-api-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.679299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.679264 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.727848 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.727955 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728083 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728239 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728342 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728430 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728479 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728533 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.728664 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\") pod \"2cd93542-5857-4474-bc8d-63f794f871cf\" (UID: \"2cd93542-5857-4474-bc8d-63f794f871cf\") " Oct 29 22:30:45.729930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.729376 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.730570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.730468 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.734779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.732002 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.734779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.732314 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:45.734779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.732601 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.734779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.732660 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.787248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.787042 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.787248 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.787211 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:45.788157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.787983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "kube-api-access-tjcfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:45.790718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.790676 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo" pod "2cd93542-5857-4474-bc8d-63f794f871cf" (UID: "2cd93542-5857-4474-bc8d-63f794f871cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832306 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832342 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832356 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832371 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd93542-5857-4474-bc8d-63f794f871cf-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832384 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832397 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd93542-5857-4474-bc8d-63f794f871cf-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832410 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcfd\" (UniqueName: \"kubernetes.io/projected/2cd93542-5857-4474-bc8d-63f794f871cf-kube-api-access-tjcfd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.832451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:45.832424 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd93542-5857-4474-bc8d-63f794f871cf-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:46.048925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.040095 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 22:30:46.080747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.080710 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" containerID="cri-o://8341632414bb22448207390f02512acf413b809ade6cb052d211209ed5963bf7" gracePeriod=30 Oct 29 22:30:46.125967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.125560 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" event={"ID":"ec1c4d47-4531-49d1-bbd1-739339754669","Type":"ContainerDied","Data":"1be161047a13f00f7533d0d72baf7a7ef26bcb5752f97d9b7a8a95d8ea5665e1"} Oct 29 22:30:46.125967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.125636 4376 scope.go:122] "RemoveContainer" containerID="cd864efddc677ed26bec029fc60442842532ccf7a3f1e55f9b79b6f2597f4419" Oct 29 22:30:46.125967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.125784 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2" Oct 29 22:30:46.129511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.129482 4376 generic.go:356] "Generic (PLEG): container finished" podID="2cd93542-5857-4474-bc8d-63f794f871cf" containerID="9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3" exitCode=0 Oct 29 22:30:46.129651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.129549 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" Oct 29 22:30:46.129719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.129668 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" event={"ID":"2cd93542-5857-4474-bc8d-63f794f871cf","Type":"ContainerDied","Data":"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3"} Oct 29 22:30:46.129773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.129715 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p" event={"ID":"2cd93542-5857-4474-bc8d-63f794f871cf","Type":"ContainerDied","Data":"f2b8d3cefcbc8dec8067c8288861cea746a17fd43c297f2543d29a31fa6a2b3c"} Oct 29 22:30:46.134959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.132788 4376 generic.go:356] "Generic (PLEG): container finished" podID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerID="d290e41260584899e16f00ca6b124f928cf6bf75a3d29219b7c38b2f52cbf1db" exitCode=0 Oct 29 22:30:46.134959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.132982 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerDied","Data":"d290e41260584899e16f00ca6b124f928cf6bf75a3d29219b7c38b2f52cbf1db"} Oct 29 22:30:46.134959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.133543 4376 scope.go:122] "RemoveContainer" containerID="d290e41260584899e16f00ca6b124f928cf6bf75a3d29219b7c38b2f52cbf1db" Oct 29 22:30:46.139304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.137779 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:46.139304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.137871 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:46.139304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.137935 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:30:46.153313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.151419 4376 scope.go:122] "RemoveContainer" containerID="9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3" Oct 29 22:30:46.184846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.184817 4376 scope.go:122] "RemoveContainer" containerID="9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3" Oct 29 22:30:46.185460 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:46.185433 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3\": container with ID starting with 9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3 not found: ID does not exist" containerID="9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3" Oct 29 22:30:46.185547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.185469 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3"} err="failed to get container status \"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3\": rpc error: code = NotFound desc = could not find container \"9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3\": container with ID starting with 9f73729d096a4eb114656ea12e43f6205896bf4d25bc67dfa9290451ed5a20d3 not found: ID does not exist" Oct 29 22:30:46.273213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.272942 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:46.280132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.280102 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888550-vrnp2"] Oct 29 22:30:46.310063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.309980 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:30:46.312715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.312681 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:30:46.312855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.312772 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:30:46.316834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:46.316808 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-5mf9p"] Oct 29 22:30:46.436522 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:30:46.436480 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:30:47.168686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.168646 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerStarted","Data":"0eb410e3ccbc8e4f2189b81dab21eee741e9979829f570d2247bf9c25f46fcf1"} Oct 29 22:30:47.274677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.274642 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" path="/var/lib/kubelet/pods/2cd93542-5857-4474-bc8d-63f794f871cf/volumes" Oct 29 22:30:47.275253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.275237 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" path="/var/lib/kubelet/pods/ec1c4d47-4531-49d1-bbd1-739339754669/volumes" Oct 29 22:30:47.515844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.515808 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:30:47.516092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.516044 4376 patch_prober.go:29] interesting pod/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 29 22:30:47.516241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:47.516127 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 29 22:30:48.516470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:48.516433 4376 patch_prober.go:29] interesting pod/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 29 22:30:48.516884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:48.516492 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 29 22:30:49.516136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:49.516103 4376 patch_prober.go:29] interesting pod/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 29 22:30:49.516321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:49.516153 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 29 22:30:50.516759 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:50.516723 4376 patch_prober.go:29] interesting pod/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 29 22:30:50.517278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:50.516782 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 29 22:30:51.261973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.261937 4376 generic.go:356] "Generic (PLEG): container finished" podID="17317361-6678-4683-9202-e15636063a6b" containerID="8341632414bb22448207390f02512acf413b809ade6cb052d211209ed5963bf7" exitCode=0 Oct 29 22:30:51.275927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.262406 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerDied","Data":"8341632414bb22448207390f02512acf413b809ade6cb052d211209ed5963bf7"} Oct 29 22:30:51.276320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.276297 4376 scope.go:122] "RemoveContainer" containerID="5b6b270d46863b583ed577d3f86f979613b612771a08acd6fa13632aa0ae45b9" Oct 29 22:30:51.520278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.520244 4376 patch_prober.go:29] interesting pod/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 29 22:30:51.520882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.520851 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 29 22:30:51.929151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.929116 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988533 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988613 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988658 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988688 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988796 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988837 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.988957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.989148 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.992998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.989261 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\") pod \"17317361-6678-4683-9202-e15636063a6b\" (UID: \"17317361-6678-4683-9202-e15636063a6b\") " Oct 29 22:30:51.994540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.994505 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:51.994898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.994880 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:51.997293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.996172 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:51.997293 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.997254 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:51.997799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:51.997779 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:52.001013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.000898 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:52.001957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.001925 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:52.002202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.002173 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "kube-api-access-6qnmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:52.009689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.009646 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy" pod "17317361-6678-4683-9202-e15636063a6b" (UID: "17317361-6678-4683-9202-e15636063a6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:52.093116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093075 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093116 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093131 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17317361-6678-4683-9202-e15636063a6b-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093144 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093158 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qnmj\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-kube-api-access-6qnmj\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093173 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17317361-6678-4683-9202-e15636063a6b-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093185 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093198 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17317361-6678-4683-9202-e15636063a6b-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.093435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.093210 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17317361-6678-4683-9202-e15636063a6b-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:52.273946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.273880 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" Oct 29 22:30:52.274103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.273918 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj" event={"ID":"17317361-6678-4683-9202-e15636063a6b","Type":"ContainerDied","Data":"d7904de52b2f04f1d5b8fb0dfb1506332dfee9fbfa3285b66f5250ee54a9f51a"} Oct 29 22:30:52.274103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.274057 4376 scope.go:122] "RemoveContainer" containerID="8341632414bb22448207390f02512acf413b809ade6cb052d211209ed5963bf7" Oct 29 22:30:52.333056 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.333018 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 22:30:52.345471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.345440 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-jqhkj"] Oct 29 22:30:52.521623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.521588 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:30:52.522138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.522117 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:30:52.522764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:52.522741 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:30:53.275143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:53.275111 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17317361-6678-4683-9202-e15636063a6b" path="/var/lib/kubelet/pods/17317361-6678-4683-9202-e15636063a6b/volumes" Oct 29 22:30:57.927473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.927441 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:30:57.928354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928335 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928356 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928369 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928374 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928385 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928390 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928396 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.928411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928401 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928414 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928418 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928426 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928430 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928437 4376 cpu_manager.go:414] "containerMap: removing container" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928440 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928462 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928466 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928484 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928488 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928498 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928504 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928513 4376 cpu_manager.go:414] "containerMap: removing container" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928517 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928528 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.928626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928534 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928639 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928648 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928655 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928663 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928669 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928677 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928683 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928688 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd93542-5857-4474-bc8d-63f794f871cf" containerName="istio-proxy" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928696 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928704 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928711 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:57.929126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.928718 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:57.934298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.934272 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:57.955814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.955787 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:30:57.962019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.961763 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-90f46c\"" Oct 29 22:30:57.962019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:57.961834 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-90f46c-dockercfg-dnjx5\"" Oct 29 22:30:58.021252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.021222 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:30:58.022098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022082 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:58.022160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022099 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fed219-8197-459a-827f-7e6e63438780" containerName="cleanup" Oct 29 22:30:58.022160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022122 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:58.022160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022127 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:58.022160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022147 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:58.022160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022153 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedada67-d57f-476a-ba7f-6a9204c1f379" containerName="cleanup" Oct 29 22:30:58.022370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022281 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1bfd34-89b1-4041-8eaa-9e22f9e4ded4" containerName="cleanup" Oct 29 22:30:58.022370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022289 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1c4d47-4531-49d1-bbd1-739339754669" containerName="cleanup" Oct 29 22:30:58.022370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.022294 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="17317361-6678-4683-9202-e15636063a6b" containerName="istio-proxy" Oct 29 22:30:58.026102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.026082 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.040709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.040676 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.041036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.041008 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8g9g\" (UniqueName: \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.045392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.045371 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:30:58.172039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.167564 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.172039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.167693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8g9g\" (UniqueName: \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.172039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.167786 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2p7g\" (UniqueName: \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.172039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.167945 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.172039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.168936 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.273880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.273832 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.274315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.274290 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2p7g\" (UniqueName: \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.275880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.275858 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.276594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.276576 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8g9g\" (UniqueName: \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\") pod \"payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.345581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.345542 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2p7g\" (UniqueName: \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\") pod \"payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.346278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.346254 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:58.552082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.551206 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:58.739745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.739715 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:30:58.740396 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:30:58.740363 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea6b72b_dd47_4ce5_af72_3b3f9b90f699.slice/crio-93aab8924da339c4357b97fc90753a766c6f5ef359b52cdca73793d71efb1cc7 WatchSource:0}: Error finding container 93aab8924da339c4357b97fc90753a766c6f5ef359b52cdca73793d71efb1cc7: Status 404 returned error can't find the container with id 93aab8924da339c4357b97fc90753a766c6f5ef359b52cdca73793d71efb1cc7 Oct 29 22:30:58.879452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:58.879423 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:30:59.486923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.483546 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" event={"ID":"63ec2e5a-2fc1-434a-a023-24016a6333d0","Type":"ContainerStarted","Data":"916dd924fef9b93a7f19d73f063260258ee91e93461d9ef925311896c4f34235"} Oct 29 22:30:59.486923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.483615 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" event={"ID":"63ec2e5a-2fc1-434a-a023-24016a6333d0","Type":"ContainerStarted","Data":"2a15e1da2d8e1a4605515ae85676c5c33be8cc847be47280db1c0adb4164d10c"} Oct 29 22:30:59.486923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.484941 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:30:59.499428 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.499380 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" event={"ID":"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699","Type":"ContainerStarted","Data":"6801cd90e55913e82aa9fea753141071874c2cbfc7d01ba5e8a03e1211ff03d9"} Oct 29 22:30:59.499723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.499685 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" event={"ID":"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699","Type":"ContainerStarted","Data":"93aab8924da339c4357b97fc90753a766c6f5ef359b52cdca73793d71efb1cc7"} Oct 29 22:30:59.515379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.515352 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:30:59.637930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.637375 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" podStartSLOduration=2.6373542949999997 podStartE2EDuration="2.637354295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:59.626288976 +0000 UTC m=+2918.474765832" watchObservedRunningTime="2026-10-29 22:30:59.637354295 +0000 UTC m=+2918.485831140" Oct 29 22:30:59.642950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:30:59.641493 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" podStartSLOduration=2.641479043 podStartE2EDuration="2.641479043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:59.5629895 +0000 UTC m=+2918.411466346" watchObservedRunningTime="2026-10-29 22:30:59.641479043 +0000 UTC m=+2918.489955904" Oct 29 22:31:07.032757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.032678 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:31:07.037858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.037840 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.055603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.055576 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:31:07.060875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.060837 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgwr\" (UniqueName: \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.061043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.061028 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.064028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.064002 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-26abe8\"" Oct 29 22:31:07.064028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.064001 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-26abe8-dockercfg-lp275\"" Oct 29 22:31:07.196939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.194601 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgwr\" (UniqueName: \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.196939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.194758 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.241932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.241755 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.409094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.409013 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgwr\" (UniqueName: \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\") pod \"payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.656947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.656917 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:07.970433 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:31:07.970397 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3870f7_ff9c_4408_9deb_9b2a09ec24fd.slice/crio-2b9ed1c7b991b50f524e13828712dde5e91227e052249b4bdd1f96757225fef4 WatchSource:0}: Error finding container 2b9ed1c7b991b50f524e13828712dde5e91227e052249b4bdd1f96757225fef4: Status 404 returned error can't find the container with id 2b9ed1c7b991b50f524e13828712dde5e91227e052249b4bdd1f96757225fef4 Oct 29 22:31:07.970944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:07.970893 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:31:08.578077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:08.578040 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" event={"ID":"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd","Type":"ContainerStarted","Data":"13a036d757805ad8d427b0366ad7fd71fb1df90cbf43f3228782006a5c07141f"} Oct 29 22:31:08.578545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:08.578101 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:08.578545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:08.578122 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" event={"ID":"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd","Type":"ContainerStarted","Data":"2b9ed1c7b991b50f524e13828712dde5e91227e052249b4bdd1f96757225fef4"} Oct 29 22:31:08.663768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:08.652736 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" podStartSLOduration=1.652708581 podStartE2EDuration="1.652708581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:08.640209724 +0000 UTC m=+2927.488686564" watchObservedRunningTime="2026-10-29 22:31:08.652708581 +0000 UTC m=+2927.501185472" Oct 29 22:31:11.519795 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:11.519458 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:31:11.521045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:11.521019 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:31:13.998946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:13.998896 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:31:13.999344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:13.999113 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" containerID="cri-o://374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64" gracePeriod=30 Oct 29 22:31:14.013738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:14.013709 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:31:14.013932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:14.013891 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" containerID="cri-o://f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0" gracePeriod=30 Oct 29 22:31:16.435082 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:16.435042 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:31:19.301080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.301051 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:31:19.367354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.367327 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:31:19.453211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453168 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.453211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453225 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453249 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453296 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453375 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453405 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453474 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453479 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453507 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453552 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453648 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453710 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.453824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453783 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453861 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\") pod \"53e70e49-a950-43e9-b72f-7a8f2afa4200\" (UID: \"53e70e49-a950-43e9-b72f-7a8f2afa4200\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.453895 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.454006 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.454039 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.454150 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.454201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.454195 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\") pod \"0a199270-753e-4d4e-8e66-306cec47549d\" (UID: \"0a199270-753e-4d4e-8e66-306cec47549d\") " Oct 29 22:31:19.455213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.455185 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:19.456074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.455659 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e70e49-a950-43e9-b72f-7a8f2afa4200-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.456074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.455688 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.456074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.455701 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.456074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.455727 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.456473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.456209 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:19.456473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.456447 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.456735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.456712 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:19.456959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.456929 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.456959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.456944 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.457139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.457003 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.457218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.457154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.457493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.457471 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:31:19.457621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.457601 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.459388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.459364 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:19.459875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.459810 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:31:19.459875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.459842 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:31:19.459875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.459865 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh" pod "0a199270-753e-4d4e-8e66-306cec47549d" (UID: "0a199270-753e-4d4e-8e66-306cec47549d"). InnerVolumeSpecName "kube-api-access-dhmdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:19.461075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.461055 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt" pod "53e70e49-a950-43e9-b72f-7a8f2afa4200" (UID: "53e70e49-a950-43e9-b72f-7a8f2afa4200"). InnerVolumeSpecName "kube-api-access-zq4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557642 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmdh\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-kube-api-access-dhmdh\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557697 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a199270-753e-4d4e-8e66-306cec47549d-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557712 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557720 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557735 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557747 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4gt\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-kube-api-access-zq4gt\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.557754 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557760 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557773 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557785 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557793 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a199270-753e-4d4e-8e66-306cec47549d-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557800 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557807 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a199270-753e-4d4e-8e66-306cec47549d-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557815 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e70e49-a950-43e9-b72f-7a8f2afa4200-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.558322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.557824 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a199270-753e-4d4e-8e66-306cec47549d-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:19.589118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.589093 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:31:19.683094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.682868 4376 generic.go:356] "Generic (PLEG): container finished" podID="0a199270-753e-4d4e-8e66-306cec47549d" containerID="374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64" exitCode=0 Oct 29 22:31:19.683094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.682955 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" Oct 29 22:31:19.683094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.683017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" event={"ID":"0a199270-753e-4d4e-8e66-306cec47549d","Type":"ContainerDied","Data":"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64"} Oct 29 22:31:19.683094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.683064 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8" event={"ID":"0a199270-753e-4d4e-8e66-306cec47549d","Type":"ContainerDied","Data":"29b64b949727f2f006dbd8f79d8c43af1766f44d659d040105c4fc819b37bca6"} Oct 29 22:31:19.683094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.683094 4376 scope.go:122] "RemoveContainer" containerID="374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64" Oct 29 22:31:19.685633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.685607 4376 generic.go:356] "Generic (PLEG): container finished" podID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerID="f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0" exitCode=0 Oct 29 22:31:19.685852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.685837 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" event={"ID":"53e70e49-a950-43e9-b72f-7a8f2afa4200","Type":"ContainerDied","Data":"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0"} Oct 29 22:31:19.685930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.685873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" event={"ID":"53e70e49-a950-43e9-b72f-7a8f2afa4200","Type":"ContainerDied","Data":"f6e110748d3c30eeb19332d93ce63e7492d2998576c0db56f8df93941d099fc2"} Oct 29 22:31:19.686102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.686090 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl" Oct 29 22:31:19.699003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.698961 4376 scope.go:122] "RemoveContainer" containerID="374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64" Oct 29 22:31:19.699275 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:19.699256 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64\": container with ID starting with 374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64 not found: ID does not exist" containerID="374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64" Oct 29 22:31:19.699366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.699283 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64"} err="failed to get container status \"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64\": rpc error: code = NotFound desc = could not find container \"374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64\": container with ID starting with 374c8338d60bbde460d55bc2a671eb75aee08eb288fdb90c7ba19358d2568d64 not found: ID does not exist" Oct 29 22:31:19.699366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.699300 4376 scope.go:122] "RemoveContainer" containerID="f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0" Oct 29 22:31:19.710333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.710316 4376 scope.go:122] "RemoveContainer" containerID="f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0" Oct 29 22:31:19.710608 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:19.710583 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0\": container with ID starting with f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0 not found: ID does not exist" containerID="f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0" Oct 29 22:31:19.710665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.710618 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0"} err="failed to get container status \"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0\": rpc error: code = NotFound desc = could not find container \"f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0\": container with ID starting with f74ceaeaa808ed36dfd2e1ce9ed9ec82d97ad6119ea760bbaccc42ac3e3e23c0 not found: ID does not exist" Oct 29 22:31:19.754771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.754737 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:31:19.759074 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.759049 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-z7ql8"] Oct 29 22:31:19.792835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.792809 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:31:19.798194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:19.798171 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c7lfl"] Oct 29 22:31:21.276560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:21.276525 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a199270-753e-4d4e-8e66-306cec47549d" path="/var/lib/kubelet/pods/0a199270-753e-4d4e-8e66-306cec47549d/volumes" Oct 29 22:31:21.277121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:21.277104 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" path="/var/lib/kubelet/pods/53e70e49-a950-43e9-b72f-7a8f2afa4200/volumes" Oct 29 22:31:28.537842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.537802 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:28.539006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.538980 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" Oct 29 22:31:28.539006 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.539004 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" Oct 29 22:31:28.539212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.539048 4376 cpu_manager.go:414] "containerMap: removing container" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" Oct 29 22:31:28.539212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.539056 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" Oct 29 22:31:28.539317 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.539290 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a199270-753e-4d4e-8e66-306cec47549d" containerName="istio-proxy" Oct 29 22:31:28.539317 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.539314 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e70e49-a950-43e9-b72f-7a8f2afa4200" containerName="istio-proxy" Oct 29 22:31:28.545153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.545129 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.560843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.560819 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:28.570712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.568300 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"routing-overlay\"" Oct 29 22:31:28.570712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.568308 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"agc-cred-010cbd9a21cf\"" Oct 29 22:31:28.570712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.568361 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"openshift-service-ca.crt\"" Oct 29 22:31:28.570712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.568436 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a\"" Oct 29 22:31:28.570712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.568492 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"kube-root-ca.crt\"" Oct 29 22:31:28.573562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.573542 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-e2e--95fdccdd-dockercfg-542wd\"" Oct 29 22:31:28.602109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.602076 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.602262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.602245 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs87\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.602322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.602303 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.602463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.602437 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.703532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.703494 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.703685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.703585 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs87\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.703685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.703623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.703685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.703669 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.704292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.704267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.704414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.704394 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.706286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.706261 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.728684 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.728657 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs87\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:28.861862 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:28.861763 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:29.182247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.182211 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:29.182766 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:31:29.182741 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc573113_2b00_427e_acd1_510579de6adc.slice/crio-f9469c2447ad13ae2d67a4878b53a6e32020429a98605a0eb0769385af985acd WatchSource:0}: Error finding container f9469c2447ad13ae2d67a4878b53a6e32020429a98605a0eb0769385af985acd: Status 404 returned error can't find the container with id f9469c2447ad13ae2d67a4878b53a6e32020429a98605a0eb0769385af985acd Oct 29 22:31:29.282332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.282302 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:29.809934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.809460 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" event={"ID":"fc573113-2b00-427e-acd1-510579de6adc","Type":"ContainerStarted","Data":"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb"} Oct 29 22:31:29.809934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.809531 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" event={"ID":"fc573113-2b00-427e-acd1-510579de6adc","Type":"ContainerStarted","Data":"f9469c2447ad13ae2d67a4878b53a6e32020429a98605a0eb0769385af985acd"} Oct 29 22:31:29.819925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.816241 4376 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" secret="" err="secret \"payload-processing-external-model-e2e--95fdccdd-dockercfg-542wd\" not found" Oct 29 22:31:29.819925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:29.817386 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:29.866927 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:29.866033 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0c3d9a/agc-cred-010cbd9a21cf: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:29.866927 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:29.866081 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:29.866927 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:29.866170 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:30.36612642 +0000 UTC m=+2949.214603262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:29.870926 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:29.867412 4376 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:29.870926 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:29.867477 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:30.367461091 +0000 UTC m=+2949.215937926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:30.373298 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:30.373262 4376 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:30.373298 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:30.373292 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0c3d9a/agc-cred-010cbd9a21cf: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:30.373606 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:30.373312 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:30.373606 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:30.373342 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:31.373323685 +0000 UTC m=+2950.221800525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:30.373606 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:30.373383 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:31.373363961 +0000 UTC m=+2950.221840788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:30.866163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:30.866109 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" podStartSLOduration=2.866093783 podStartE2EDuration="2.866093783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:29.86771165 +0000 UTC m=+2948.716188496" watchObservedRunningTime="2026-10-29 22:31:30.866093783 +0000 UTC m=+2949.714570627" Oct 29 22:31:31.389579 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:31.389549 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0c3d9a/agc-cred-010cbd9a21cf: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:31.389579 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:31.389582 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:31.389843 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:31.389545 4376 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:31.389843 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:31.389640 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:33.389622828 +0000 UTC m=+2952.238099662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:31.389843 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:31.389711 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:33.389698542 +0000 UTC m=+2952.238175372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:33.424343 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:33.424306 4376 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:33.424819 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:33.424382 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:37.424366654 +0000 UTC m=+2956.272843490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a" not found Oct 29 22:31:33.424819 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:33.424410 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0c3d9a/agc-cred-010cbd9a21cf: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:33.424819 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:33.424431 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5: secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:33.424819 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:33.424507 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials podName:fc573113-2b00-427e-acd1-510579de6adc nodeName:}" failed. No retries permitted until 2026-10-29 22:31:37.424487605 +0000 UTC m=+2956.272964436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" (UID: "fc573113-2b00-427e-acd1-510579de6adc") : secret "agc-cred-010cbd9a21cf" not found Oct 29 22:31:35.819069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:35.819022 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" podUID="fc573113-2b00-427e-acd1-510579de6adc" containerName="payload-processing" containerID="cri-o://bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb" gracePeriod=25 Oct 29 22:31:36.106504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.106476 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:36.169939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.169853 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\") pod \"fc573113-2b00-427e-acd1-510579de6adc\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " Oct 29 22:31:36.170097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.170067 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\") pod \"fc573113-2b00-427e-acd1-510579de6adc\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " Oct 29 22:31:36.170146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.170109 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\") pod \"fc573113-2b00-427e-acd1-510579de6adc\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " Oct 29 22:31:36.170419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.170396 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\") pod \"fc573113-2b00-427e-acd1-510579de6adc\" (UID: \"fc573113-2b00-427e-acd1-510579de6adc\") " Oct 29 22:31:36.170543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.170427 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume" pod "fc573113-2b00-427e-acd1-510579de6adc" (UID: "fc573113-2b00-427e-acd1-510579de6adc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:36.170650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.170629 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume" pod "fc573113-2b00-427e-acd1-510579de6adc" (UID: "fc573113-2b00-427e-acd1-510579de6adc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:36.171481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.171290 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:36.171481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.171337 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc573113-2b00-427e-acd1-510579de6adc-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:36.172774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.172748 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87" pod "fc573113-2b00-427e-acd1-510579de6adc" (UID: "fc573113-2b00-427e-acd1-510579de6adc"). InnerVolumeSpecName "kube-api-access-zcs87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:36.173152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.173127 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials" pod "fc573113-2b00-427e-acd1-510579de6adc" (UID: "fc573113-2b00-427e-acd1-510579de6adc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:36.273289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.273259 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcs87\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-kube-api-access-zcs87\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:36.273289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.273286 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc573113-2b00-427e-acd1-510579de6adc-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:36.865467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.865431 4376 generic.go:356] "Generic (PLEG): container finished" podID="fc573113-2b00-427e-acd1-510579de6adc" containerID="bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb" exitCode=0 Oct 29 22:31:36.865467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.865466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" Oct 29 22:31:36.865960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.865504 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" event={"ID":"fc573113-2b00-427e-acd1-510579de6adc","Type":"ContainerDied","Data":"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb"} Oct 29 22:31:36.865960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.865561 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5" event={"ID":"fc573113-2b00-427e-acd1-510579de6adc","Type":"ContainerDied","Data":"f9469c2447ad13ae2d67a4878b53a6e32020429a98605a0eb0769385af985acd"} Oct 29 22:31:36.865960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.865590 4376 scope.go:122] "RemoveContainer" containerID="bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb" Oct 29 22:31:36.882453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.882426 4376 scope.go:122] "RemoveContainer" containerID="bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb" Oct 29 22:31:36.882825 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:36.882799 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb\": container with ID starting with bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb not found: ID does not exist" containerID="bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb" Oct 29 22:31:36.882957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.882837 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb"} err="failed to get container status \"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb\": rpc error: code = NotFound desc = could not find container \"bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb\": container with ID starting with bdd6eaf1f15638c30cd69f3cc1597699914e28b0fd3bf5f9b53699343f426ceb not found: ID does not exist" Oct 29 22:31:36.925413 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.925381 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:36.937066 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:36.937036 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-6879642r5"] Oct 29 22:31:37.274417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:37.274381 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc573113-2b00-427e-acd1-510579de6adc" path="/var/lib/kubelet/pods/fc573113-2b00-427e-acd1-510579de6adc/volumes" Oct 29 22:31:44.934727 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.934686 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:31:44.936283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.936251 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fc573113-2b00-427e-acd1-510579de6adc" containerName="payload-processing" Oct 29 22:31:44.936283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.936284 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc573113-2b00-427e-acd1-510579de6adc" containerName="payload-processing" Oct 29 22:31:44.936527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.936511 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc573113-2b00-427e-acd1-510579de6adc" containerName="payload-processing" Oct 29 22:31:44.942507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.942477 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.964700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.964668 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:31:44.970134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.970099 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d09411-openshift-default-dockercfg-rwwzk\"" Oct 29 22:31:44.985856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.985803 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.986204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.986183 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.986511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.986451 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.986836 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.986813 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.987027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.987010 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.987154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.987060 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.987253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.987234 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kg5\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.987400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.987380 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:44.987499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:44.987413 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.089701 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kg5\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.089815 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.089842 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.089870 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.092317 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.092405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.092613 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.092713 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.093691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.092762 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.094408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.094187 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.094478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.094450 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.094709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.094679 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.106078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.094940 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.106078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.095856 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.114984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.112554 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.161305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.148001 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kg5\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.161305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.148545 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.161305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.149051 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.265886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.265851 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:45.702595 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:31:45.702566 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51391134_c2b5_4bdf_b54f_971a421cfcaf.slice/crio-11e7d355184efdd7bf81c08f6b1adbca8389e785ddcd4d557cd5db3c5b17c123 WatchSource:0}: Error finding container 11e7d355184efdd7bf81c08f6b1adbca8389e785ddcd4d557cd5db3c5b17c123: Status 404 returned error can't find the container with id 11e7d355184efdd7bf81c08f6b1adbca8389e785ddcd4d557cd5db3c5b17c123 Oct 29 22:31:45.702885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.702861 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:31:45.705734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.705695 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:45.705853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.705804 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:45.705853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:45.705843 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:31:46.042964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.042886 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerStarted","Data":"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88"} Oct 29 22:31:46.042964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.042964 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerStarted","Data":"11e7d355184efdd7bf81c08f6b1adbca8389e785ddcd4d557cd5db3c5b17c123"} Oct 29 22:31:46.104682 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.104628 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podStartSLOduration=2.104614648 podStartE2EDuration="2.104614648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:46.089551159 +0000 UTC m=+2964.938028031" watchObservedRunningTime="2026-10-29 22:31:46.104614648 +0000 UTC m=+2964.953091492" Oct 29 22:31:46.267261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.267231 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:46.269654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.269626 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:31:46.269817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:46.269672 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:31:46.435421 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:31:46.435334 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:31:47.277511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:47.277479 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:48.059463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:48.059427 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:48.060595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:48.060568 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:31:50.532514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.532475 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:31:50.537607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.537580 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.555790 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.555755 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:31:50.562887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.562860 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d09411-dockercfg-ckvgk\"" Oct 29 22:31:50.576841 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.576716 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d09411\"" Oct 29 22:31:50.770072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.770018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.770531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.770501 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdm4\" (UniqueName: \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.861078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.857970 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:31:50.889659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.889617 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.890153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.890126 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdm4\" (UniqueName: \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.891892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.891866 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.892624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.892606 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:50.924286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.917482 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdm4\" (UniqueName: \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\") pod \"payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:50.924286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.922247 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:31:50.992015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.991979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:50.992204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:50.992039 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2mr\" (UniqueName: \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.093381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.093338 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.093629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.093402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2mr\" (UniqueName: \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.094930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.094868 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.119252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.119156 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2mr\" (UniqueName: \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\") pod \"payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.156236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.156201 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:51.228351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.228317 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:51.476889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.476846 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:31:51.477274 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:31:51.477248 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe560f2c_f950_48ba_97d8_4f2a112d590c.slice/crio-f9edb909f332744236c1fb3f7dd0eb9b5c881b89093e2b99bc78580bccb568ae WatchSource:0}: Error finding container f9edb909f332744236c1fb3f7dd0eb9b5c881b89093e2b99bc78580bccb568ae: Status 404 returned error can't find the container with id f9edb909f332744236c1fb3f7dd0eb9b5c881b89093e2b99bc78580bccb568ae Oct 29 22:31:51.566990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:51.566957 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:31:51.568517 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:31:51.568487 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3443574_dc80_4ca9_b8c1_8c052afaf411.slice/crio-2352af2fbc0be2fa8d78371ee91fef623a223f6ab416f92b409b638c415b7fd2 WatchSource:0}: Error finding container 2352af2fbc0be2fa8d78371ee91fef623a223f6ab416f92b409b638c415b7fd2: Status 404 returned error can't find the container with id 2352af2fbc0be2fa8d78371ee91fef623a223f6ab416f92b409b638c415b7fd2 Oct 29 22:31:52.092048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.092008 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" event={"ID":"c3443574-dc80-4ca9-b8c1-8c052afaf411","Type":"ContainerStarted","Data":"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4"} Oct 29 22:31:52.092279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.092075 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" event={"ID":"c3443574-dc80-4ca9-b8c1-8c052afaf411","Type":"ContainerStarted","Data":"2352af2fbc0be2fa8d78371ee91fef623a223f6ab416f92b409b638c415b7fd2"} Oct 29 22:31:52.092279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.092105 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:31:52.093916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.093872 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" event={"ID":"be560f2c-f950-48ba-97d8-4f2a112d590c","Type":"ContainerStarted","Data":"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad"} Oct 29 22:31:52.094065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.093933 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" event={"ID":"be560f2c-f950-48ba-97d8-4f2a112d590c","Type":"ContainerStarted","Data":"f9edb909f332744236c1fb3f7dd0eb9b5c881b89093e2b99bc78580bccb568ae"} Oct 29 22:31:52.094065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.093960 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:31:52.162933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.162563 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" podStartSLOduration=2.162538743 podStartE2EDuration="2.162538743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:52.152295765 +0000 UTC m=+2971.000772611" watchObservedRunningTime="2026-10-29 22:31:52.162538743 +0000 UTC m=+2971.011015588" Oct 29 22:31:52.198942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:31:52.198867 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" podStartSLOduration=2.198851278 podStartE2EDuration="2.198851278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:52.193786459 +0000 UTC m=+2971.042263307" watchObservedRunningTime="2026-10-29 22:31:52.198851278 +0000 UTC m=+2971.047328182" Oct 29 22:32:03.105707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:03.105676 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:32:03.106218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:03.105764 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:32:11.275834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:11.275799 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:32:11.276318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:11.275999 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" podUID="4b901937-83eb-40c9-9532-07223880676f" containerName="sim" containerID="cri-o://971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a" gracePeriod=30 Oct 29 22:32:12.062686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.062652 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:32:12.136695 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.136482 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\" (UniqueName: \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\") pod \"4b901937-83eb-40c9-9532-07223880676f\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " Oct 29 22:32:12.137226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.137198 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\" (UniqueName: \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\") pod \"4b901937-83eb-40c9-9532-07223880676f\" (UID: \"4b901937-83eb-40c9-9532-07223880676f\") " Oct 29 22:32:12.148229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.148193 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf" pod "4b901937-83eb-40c9-9532-07223880676f" (UID: "4b901937-83eb-40c9-9532-07223880676f"). InnerVolumeSpecName "kube-api-access-tkwpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:12.175873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.175683 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls" pod "4b901937-83eb-40c9-9532-07223880676f" (UID: "4b901937-83eb-40c9-9532-07223880676f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:12.241787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.241748 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwpf\" (UniqueName: \"kubernetes.io/projected/4b901937-83eb-40c9-9532-07223880676f-kube-api-access-tkwpf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:12.241787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.241782 4376 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4b901937-83eb-40c9-9532-07223880676f-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:12.339946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.339802 4376 generic.go:356] "Generic (PLEG): container finished" podID="4b901937-83eb-40c9-9532-07223880676f" containerID="971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a" exitCode=0 Oct 29 22:32:12.339946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.339845 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" Oct 29 22:32:12.339946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.339882 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" event={"ID":"4b901937-83eb-40c9-9532-07223880676f","Type":"ContainerDied","Data":"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a"} Oct 29 22:32:12.340596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.339959 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2" event={"ID":"4b901937-83eb-40c9-9532-07223880676f","Type":"ContainerDied","Data":"f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8"} Oct 29 22:32:12.340596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.339977 4376 scope.go:122] "RemoveContainer" containerID="971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a" Oct 29 22:32:12.356308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.356073 4376 scope.go:122] "RemoveContainer" containerID="971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a" Oct 29 22:32:12.357062 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:12.357031 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a\": container with ID starting with 971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a not found: ID does not exist" containerID="971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a" Oct 29 22:32:12.357189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.357071 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a"} err="failed to get container status \"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a\": rpc error: code = NotFound desc = could not find container \"971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a\": container with ID starting with 971dc5443a2e0ad8bd63b68ed61f301224fa222adfacf349ecfbe166274dcf6a not found: ID does not exist" Oct 29 22:32:12.402119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.402087 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:32:12.402301 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:12.402093 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b901937_83eb_40c9_9532_07223880676f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b901937_83eb_40c9_9532_07223880676f.slice/crio-f84945a8770188d5428a6687dd66d18a886e03a06348eb8edda320a412059dc8\": RecentStats: unable to find data in memory cache]" Oct 29 22:32:12.412621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.412584 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-4366ca/e2e-usage-sim-238088-69c4df475b-hlpd2"] Oct 29 22:32:12.821111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.821077 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:12.822051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.822032 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4b901937-83eb-40c9-9532-07223880676f" containerName="sim" Oct 29 22:32:12.822118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.822051 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b901937-83eb-40c9-9532-07223880676f" containerName="sim" Oct 29 22:32:12.822233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.822224 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b901937-83eb-40c9-9532-07223880676f" containerName="sim" Oct 29 22:32:12.827357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.827337 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:12.844517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.844466 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-96cc362d023d\"" Oct 29 22:32:12.846873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.846845 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:12.853207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853184 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-f0acd722a44c\"" Oct 29 22:32:12.853374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853190 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"kube-root-ca.crt\"" Oct 29 22:32:12.853446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853194 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"openshift-service-ca.crt\"" Oct 29 22:32:12.853446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853195 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-e2e--885c5d1d-dockercfg-58nd4\"" Oct 29 22:32:12.853446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853216 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"routing-overlay\"" Oct 29 22:32:12.853446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.853278 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e7b56d\"" Oct 29 22:32:12.953954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.953912 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:12.954138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.954067 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:12.954246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.954231 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd4j\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:12.954419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:12.954397 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.056307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.056263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.056538 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.056472 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.056620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.056544 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.056689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.056675 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd4j\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.057488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.057458 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.057632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.057463 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.059159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.059137 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.082679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.082615 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd4j\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.147922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.147877 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:13.444660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.444559 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b901937-83eb-40c9-9532-07223880676f" path="/var/lib/kubelet/pods/4b901937-83eb-40c9-9532-07223880676f/volumes" Oct 29 22:32:13.480411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:13.478647 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:14.464361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:14.464306 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" event={"ID":"99ce0ae8-690a-4c83-b860-8a33cf7da448","Type":"ContainerStarted","Data":"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab"} Oct 29 22:32:14.464831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:14.464378 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" event={"ID":"99ce0ae8-690a-4c83-b860-8a33cf7da448","Type":"ContainerStarted","Data":"60fbdfe0fe62a9d76e1a888fad27aa5cbacde5f7a91854d8bb3f2c8a85a46e28"} Oct 29 22:32:14.464831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:14.464581 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:14.510783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:14.510734 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" podStartSLOduration=2.510720232 podStartE2EDuration="2.510720232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:14.50911076 +0000 UTC m=+2993.357587627" watchObservedRunningTime="2026-10-29 22:32:14.510720232 +0000 UTC m=+2993.359197076" Oct 29 22:32:16.436280 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:16.436240 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:32:24.995238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:24.995199 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:32:25.475952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:25.475890 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:29.996523 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:29.996480 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" podUID="7986b245-6921-4551-a757-f65743396213" containerName="payload-processing" containerID="cri-o://9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa" gracePeriod=25 Oct 29 22:32:30.306021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.305988 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:32:30.406939 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.406887 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\") pod \"7986b245-6921-4551-a757-f65743396213\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " Oct 29 22:32:30.407145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.406972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\") pod \"7986b245-6921-4551-a757-f65743396213\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " Oct 29 22:32:30.407145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.407068 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\") pod \"7986b245-6921-4551-a757-f65743396213\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " Oct 29 22:32:30.407301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.407232 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\") pod \"7986b245-6921-4551-a757-f65743396213\" (UID: \"7986b245-6921-4551-a757-f65743396213\") " Oct 29 22:32:30.407565 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.407529 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume" pod "7986b245-6921-4551-a757-f65743396213" (UID: "7986b245-6921-4551-a757-f65743396213"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.407812 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.407784 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume" pod "7986b245-6921-4551-a757-f65743396213" (UID: "7986b245-6921-4551-a757-f65743396213"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.408350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.408326 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.408451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.408411 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7986b245-6921-4551-a757-f65743396213-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.410079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.410050 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials" pod "7986b245-6921-4551-a757-f65743396213" (UID: "7986b245-6921-4551-a757-f65743396213"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.410168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.410117 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l" pod "7986b245-6921-4551-a757-f65743396213" (UID: "7986b245-6921-4551-a757-f65743396213"). InnerVolumeSpecName "kube-api-access-8cd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.509724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.509691 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cd6l\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-kube-api-access-8cd6l\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.509724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.509721 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7986b245-6921-4551-a757-f65743396213-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.607339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.607244 4376 generic.go:356] "Generic (PLEG): container finished" podID="7986b245-6921-4551-a757-f65743396213" containerID="9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa" exitCode=0 Oct 29 22:32:30.607339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.607300 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" Oct 29 22:32:30.607339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.607324 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" event={"ID":"7986b245-6921-4551-a757-f65743396213","Type":"ContainerDied","Data":"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa"} Oct 29 22:32:30.607633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.607413 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9" event={"ID":"7986b245-6921-4551-a757-f65743396213","Type":"ContainerDied","Data":"bcf9c40289a99dfae6a0a6f58e165bbe2bd0ade9250c37221e6baf534a23ebe2"} Oct 29 22:32:30.607633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.607443 4376 scope.go:122] "RemoveContainer" containerID="9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa" Oct 29 22:32:30.620093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.620073 4376 scope.go:122] "RemoveContainer" containerID="9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa" Oct 29 22:32:30.620362 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:30.620341 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa\": container with ID starting with 9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa not found: ID does not exist" containerID="9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa" Oct 29 22:32:30.620433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.620373 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa"} err="failed to get container status \"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa\": rpc error: code = NotFound desc = could not find container \"9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa\": container with ID starting with 9d797e14656bb03dd082d12c62dd16f5608add18fb4815dbeb28cd35a8f2effa not found: ID does not exist" Oct 29 22:32:30.667520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.667490 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:32:30.680223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:30.680195 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-9966zw6l9"] Oct 29 22:32:31.275329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:31.275297 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7986b245-6921-4551-a757-f65743396213" path="/var/lib/kubelet/pods/7986b245-6921-4551-a757-f65743396213/volumes" Oct 29 22:32:32.401321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:32.401287 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:32:37.402326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.402235 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" podUID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" containerName="payload-processing" containerID="cri-o://f3e2d38c3c78b6f1f58000316d2240b0189692a9170739eb875c21a33ddc9b22" gracePeriod=25 Oct 29 22:32:37.693072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.685137 4376 generic.go:356] "Generic (PLEG): container finished" podID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" containerID="f3e2d38c3c78b6f1f58000316d2240b0189692a9170739eb875c21a33ddc9b22" exitCode=0 Oct 29 22:32:37.693072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.685214 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" event={"ID":"09fbef2c-d6c9-4163-abd8-134c1b8c5234","Type":"ContainerDied","Data":"f3e2d38c3c78b6f1f58000316d2240b0189692a9170739eb875c21a33ddc9b22"} Oct 29 22:32:37.843387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.843362 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:32:37.946527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.946433 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\") pod \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " Oct 29 22:32:37.946527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.946525 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\") pod \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " Oct 29 22:32:37.946769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.946631 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\") pod \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " Oct 29 22:32:37.948394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.946954 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\") pod \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\" (UID: \"09fbef2c-d6c9-4163-abd8-134c1b8c5234\") " Oct 29 22:32:37.948394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.947675 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume" pod "09fbef2c-d6c9-4163-abd8-134c1b8c5234" (UID: "09fbef2c-d6c9-4163-abd8-134c1b8c5234"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:37.948394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.947849 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:37.948394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.948351 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume" pod "09fbef2c-d6c9-4163-abd8-134c1b8c5234" (UID: "09fbef2c-d6c9-4163-abd8-134c1b8c5234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:37.964044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.961128 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr" pod "09fbef2c-d6c9-4163-abd8-134c1b8c5234" (UID: "09fbef2c-d6c9-4163-abd8-134c1b8c5234"). InnerVolumeSpecName "kube-api-access-hpxkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:37.987932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:37.980208 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials" pod "09fbef2c-d6c9-4163-abd8-134c1b8c5234" (UID: "09fbef2c-d6c9-4163-abd8-134c1b8c5234"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:38.070949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.062148 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:38.070949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.062187 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fbef2c-d6c9-4163-abd8-134c1b8c5234-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:38.070949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.062204 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxkr\" (UniqueName: \"kubernetes.io/projected/09fbef2c-d6c9-4163-abd8-134c1b8c5234-kube-api-access-hpxkr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.080344 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2"] Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.081791 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7986b245-6921-4551-a757-f65743396213" containerName="payload-processing" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.081818 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7986b245-6921-4551-a757-f65743396213" containerName="payload-processing" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.081883 4376 cpu_manager.go:414] "containerMap: removing container" podUID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" containerName="payload-processing" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.081892 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" containerName="payload-processing" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.082119 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" containerName="payload-processing" Oct 29 22:32:38.097926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.082158 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7986b245-6921-4551-a757-f65743396213" containerName="payload-processing" Oct 29 22:32:38.128699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.105736 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2"] Oct 29 22:32:38.128699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.105876 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.128699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.124866 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"openshift-service-ca.crt\"" Oct 29 22:32:38.129050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.128937 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:32:38.129095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.129060 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-31abe10e24c1\"" Oct 29 22:32:38.129219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.129195 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-e2e--b7017604-dockercfg-gkpdl\"" Oct 29 22:32:38.129311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.129295 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"kube-root-ca.crt\"" Oct 29 22:32:38.143934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.127378 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"routing-overlay\"" Oct 29 22:32:38.268162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.268121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.268486 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.268464 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxdx\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.268644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.268607 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.268710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.268699 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.369958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.369892 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxdx\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.370148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.370048 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.370148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.370117 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.370273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.370200 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.370273 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.370242 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0412c3/agc-cred-31abe10e24c1: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:38.370273 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.370264 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:38.370416 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.370328 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials podName:f119325c-d6ff-4964-9060-cff1283181fd nodeName:}" failed. No retries permitted until 2026-10-29 22:32:38.870311101 +0000 UTC m=+3017.718787931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" (UID: "f119325c-d6ff-4964-9060-cff1283181fd") : secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:38.370888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.370864 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.371023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.371006 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.395947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.395891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxdx\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.537035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.536958 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:38.694877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.694850 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" Oct 29 22:32:38.695058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.694877 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89" event={"ID":"09fbef2c-d6c9-4163-abd8-134c1b8c5234","Type":"ContainerDied","Data":"f037b48c83996a9c0aa3c0e9ce0e21f3b38a4f34fd736343d026ae3ed33edecd"} Oct 29 22:32:38.695058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.694965 4376 scope.go:122] "RemoveContainer" containerID="f3e2d38c3c78b6f1f58000316d2240b0189692a9170739eb875c21a33ddc9b22" Oct 29 22:32:38.753122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.753094 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:32:38.760577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.760554 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-767d74r89"] Oct 29 22:32:38.879192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:38.879110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:38.879374 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.879294 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0412c3/agc-cred-31abe10e24c1: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:38.879374 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.879358 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:38.879485 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:38.879432 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials podName:f119325c-d6ff-4964-9060-cff1283181fd nodeName:}" failed. No retries permitted until 2026-10-29 22:32:39.879411655 +0000 UTC m=+3018.727888494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" (UID: "f119325c-d6ff-4964-9060-cff1283181fd") : secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:39.325350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:39.324770 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fbef2c-d6c9-4163-abd8-134c1b8c5234" path="/var/lib/kubelet/pods/09fbef2c-d6c9-4163-abd8-134c1b8c5234/volumes" Oct 29 22:32:39.953470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:39.953427 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:39.953887 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:39.953570 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0412c3/agc-cred-31abe10e24c1: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:39.953887 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:39.953590 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2: secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:39.953887 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:39.953642 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials podName:f119325c-d6ff-4964-9060-cff1283181fd nodeName:}" failed. No retries permitted until 2026-10-29 22:32:41.953627203 +0000 UTC m=+3020.802104027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" (UID: "f119325c-d6ff-4964-9060-cff1283181fd") : secret "agc-cred-31abe10e24c1" not found Oct 29 22:32:40.505619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.505584 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2"] Oct 29 22:32:40.506114 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:40.506093 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" podUID="f119325c-d6ff-4964-9060-cff1283181fd" Oct 29 22:32:40.712115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.712087 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:40.720281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.720256 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:40.769356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.769292 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\") pod \"f119325c-d6ff-4964-9060-cff1283181fd\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " Oct 29 22:32:40.769356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.769347 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\") pod \"f119325c-d6ff-4964-9060-cff1283181fd\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " Oct 29 22:32:40.769562 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.769486 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\") pod \"f119325c-d6ff-4964-9060-cff1283181fd\" (UID: \"f119325c-d6ff-4964-9060-cff1283181fd\") " Oct 29 22:32:40.769817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.769789 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume" pod "f119325c-d6ff-4964-9060-cff1283181fd" (UID: "f119325c-d6ff-4964-9060-cff1283181fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:40.769817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.769812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume" pod "f119325c-d6ff-4964-9060-cff1283181fd" (UID: "f119325c-d6ff-4964-9060-cff1283181fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:40.770185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.770165 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.770185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.770184 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f119325c-d6ff-4964-9060-cff1283181fd-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.771530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.771508 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx" pod "f119325c-d6ff-4964-9060-cff1283181fd" (UID: "f119325c-d6ff-4964-9060-cff1283181fd"). InnerVolumeSpecName "kube-api-access-rtxdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:40.879928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:40.875978 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxdx\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-kube-api-access-rtxdx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:41.720309 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:41.720274 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2" Oct 29 22:32:41.810791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:41.810755 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2"] Oct 29 22:32:41.815141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:41.815112 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-7cfdms2d2"] Oct 29 22:32:41.902751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:41.902717 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f119325c-d6ff-4964-9060-cff1283181fd-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:42.522840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.522801 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:32:42.528602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.528578 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.550011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.549983 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:32:42.556948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.556837 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0cb13f-openshift-default-dockercfg-fh66q\"" Oct 29 22:32:42.750926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.727535 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97hp\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.752308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.752255 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.752593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.752576 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.752787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.752756 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.753004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.752984 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.753167 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.753151 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.753425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.753395 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.753671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.753652 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.754013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.753981 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.856557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.856465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97hp\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.856830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.856802 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.857037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.857018 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.857204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.857184 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.857390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.857365 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.857544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.857530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.857723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.857709 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.871054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.871016 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.871365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.871346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.872077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.872053 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.872240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.866214 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.872600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.872576 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.872725 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.867773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.873796 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.873775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.901690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.901658 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.910477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.910444 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.970607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.955983 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97hp\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:42.979395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:42.979361 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\") pod \"e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:43.147850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.147752 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:43.278813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.278772 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f119325c-d6ff-4964-9060-cff1283181fd" path="/var/lib/kubelet/pods/f119325c-d6ff-4964-9060-cff1283181fd/volumes" Oct 29 22:32:43.500988 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:32:43.500954 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff6ada8_ae62_4a60_9cc5_3d7b82c322eb.slice/crio-9289aab746bc58a9fd0d367ea157de0a1df48a99e8585d85b21cd5f73f2c9cd4 WatchSource:0}: Error finding container 9289aab746bc58a9fd0d367ea157de0a1df48a99e8585d85b21cd5f73f2c9cd4: Status 404 returned error can't find the container with id 9289aab746bc58a9fd0d367ea157de0a1df48a99e8585d85b21cd5f73f2c9cd4 Oct 29 22:32:43.502144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.502120 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:32:43.504033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.503997 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:43.504161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.504080 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:43.504161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.504129 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:43.538143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.538115 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" podUID="99ce0ae8-690a-4c83-b860-8a33cf7da448" containerName="payload-processing" containerID="cri-o://f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab" gracePeriod=25 Oct 29 22:32:43.812834 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.812806 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:43.822734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.822703 4376 generic.go:356] "Generic (PLEG): container finished" podID="99ce0ae8-690a-4c83-b860-8a33cf7da448" containerID="f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab" exitCode=0 Oct 29 22:32:43.822884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.822740 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" Oct 29 22:32:43.822884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.822787 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" event={"ID":"99ce0ae8-690a-4c83-b860-8a33cf7da448","Type":"ContainerDied","Data":"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab"} Oct 29 22:32:43.822884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.822842 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz" event={"ID":"99ce0ae8-690a-4c83-b860-8a33cf7da448","Type":"ContainerDied","Data":"60fbdfe0fe62a9d76e1a888fad27aa5cbacde5f7a91854d8bb3f2c8a85a46e28"} Oct 29 22:32:43.822884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.822869 4376 scope.go:122] "RemoveContainer" containerID="f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab" Oct 29 22:32:43.824961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.824937 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerStarted","Data":"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da"} Oct 29 22:32:43.825085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.824982 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerStarted","Data":"9289aab746bc58a9fd0d367ea157de0a1df48a99e8585d85b21cd5f73f2c9cd4"} Oct 29 22:32:43.833779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.833760 4376 scope.go:122] "RemoveContainer" containerID="f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab" Oct 29 22:32:43.834100 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:43.834081 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab\": container with ID starting with f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab not found: ID does not exist" containerID="f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab" Oct 29 22:32:43.834181 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.834106 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab"} err="failed to get container status \"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab\": rpc error: code = NotFound desc = could not find container \"f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab\": container with ID starting with f8250b82bed5f7cb58c5e7bfc71c044666d6b1a65aedbdd888790ad728445cab not found: ID does not exist" Oct 29 22:32:43.899242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.899207 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\") pod \"99ce0ae8-690a-4c83-b860-8a33cf7da448\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " Oct 29 22:32:43.899412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.899279 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\") pod \"99ce0ae8-690a-4c83-b860-8a33cf7da448\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " Oct 29 22:32:43.899412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.899360 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\") pod \"99ce0ae8-690a-4c83-b860-8a33cf7da448\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " Oct 29 22:32:43.899547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.899421 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\") pod \"99ce0ae8-690a-4c83-b860-8a33cf7da448\" (UID: \"99ce0ae8-690a-4c83-b860-8a33cf7da448\") " Oct 29 22:32:43.901619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.901373 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume" pod "99ce0ae8-690a-4c83-b860-8a33cf7da448" (UID: "99ce0ae8-690a-4c83-b860-8a33cf7da448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:43.901619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.901445 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume" pod "99ce0ae8-690a-4c83-b860-8a33cf7da448" (UID: "99ce0ae8-690a-4c83-b860-8a33cf7da448"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:43.903713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.903675 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials" pod "99ce0ae8-690a-4c83-b860-8a33cf7da448" (UID: "99ce0ae8-690a-4c83-b860-8a33cf7da448"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:43.904363 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.904342 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j" pod "99ce0ae8-690a-4c83-b860-8a33cf7da448" (UID: "99ce0ae8-690a-4c83-b860-8a33cf7da448"). InnerVolumeSpecName "kube-api-access-bvd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:43.908512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:43.908465 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podStartSLOduration=1.908451334 podStartE2EDuration="1.908451334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:43.90742456 +0000 UTC m=+3022.755901406" watchObservedRunningTime="2026-10-29 22:32:43.908451334 +0000 UTC m=+3022.756928178" Oct 29 22:32:44.002027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.001998 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:44.002027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.002023 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvd4j\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-kube-api-access-bvd4j\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:44.002027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.002033 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce0ae8-690a-4c83-b860-8a33cf7da448-provider-credentials\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:44.002327 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.002043 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce0ae8-690a-4c83-b860-8a33cf7da448-routing-overlay-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:44.148671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.148637 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:44.149843 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.149812 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:32:44.149983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.149861 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:32:44.193120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.193090 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:44.201404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:44.201379 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54zwxtz"] Oct 29 22:32:45.154453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:45.154424 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:45.274583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:45.274551 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ce0ae8-690a-4c83-b860-8a33cf7da448" path="/var/lib/kubelet/pods/99ce0ae8-690a-4c83-b860-8a33cf7da448/volumes" Oct 29 22:32:45.842337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:45.842302 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:45.843290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:45.843266 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:32:46.435727 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:32:46.435686 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:32:47.564062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.564028 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:32:47.564953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.564935 4376 cpu_manager.go:414] "containerMap: removing container" podUID="99ce0ae8-690a-4c83-b860-8a33cf7da448" containerName="payload-processing" Oct 29 22:32:47.565009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.564954 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ce0ae8-690a-4c83-b860-8a33cf7da448" containerName="payload-processing" Oct 29 22:32:47.565156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.565146 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ce0ae8-690a-4c83-b860-8a33cf7da448" containerName="payload-processing" Oct 29 22:32:47.570894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.570866 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.604371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.604330 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:32:47.616955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.610839 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-219ccc-openshift-default-dockercfg-p8dql\"" Oct 29 22:32:47.704126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704088 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704177 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704239 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704351 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704464 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704575 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704623 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704664 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.704760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.704749 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpxt\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824592 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824706 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824798 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpxt\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824896 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.824977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.824967 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.825567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.825050 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.825632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.825585 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.834923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.826189 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.854599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.853091 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.854599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.853361 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.855408 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.855106 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.892621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.892585 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.937987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.927197 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.937987 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.931614 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpxt\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:47.966056 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:47.940248 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\") pod \"e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:48.208029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.207995 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:48.705422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.705385 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:32:48.739148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.739110 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:48.739244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.739192 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:48.739244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.739221 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:32:48.936924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.936175 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerStarted","Data":"9a730037e190e90e34d916106368a90f9d170504d2819e0d47e3dadeccfe8025"} Oct 29 22:32:48.936924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.936268 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerStarted","Data":"9c1d37792f364d6588552c02534c3631316952633727b56aebf5dae0885c1a1f"} Oct 29 22:32:48.988053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:48.987948 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podStartSLOduration=1.987929192 podStartE2EDuration="1.987929192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:48.985220364 +0000 UTC m=+3027.833697210" watchObservedRunningTime="2026-10-29 22:32:48.987929192 +0000 UTC m=+3027.836406037" Oct 29 22:32:49.208387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:49.208352 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:49.210332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:49.210304 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:32:49.210464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:49.210359 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:32:50.213996 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:50.213965 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:50.955178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:50.955143 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:50.956117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:50.956095 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:32:53.465047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.465010 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:32:53.470118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.470089 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.489317 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.489286 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:32:53.496514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.496483 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-219ccc-dockercfg-8p47d\"" Oct 29 22:32:53.496696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.496577 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-219ccc\"" Oct 29 22:32:53.802529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.801157 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7lx\" (UniqueName: \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.802529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.801348 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.933020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.932976 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.933424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.933403 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7lx\" (UniqueName: \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.933716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.933696 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:53.959104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:53.959057 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7lx\" (UniqueName: \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\") pod \"payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:54.092755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:54.092665 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:54.410339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:54.410312 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:32:54.411541 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:32:54.411513 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459699a8_f82e_414b_b0df_35d24fdb08a0.slice/crio-0d10debc219df80a8cf759dd01cd0053742c81dd1103fddd67429c6179b278d0 WatchSource:0}: Error finding container 0d10debc219df80a8cf759dd01cd0053742c81dd1103fddd67429c6179b278d0: Status 404 returned error can't find the container with id 0d10debc219df80a8cf759dd01cd0053742c81dd1103fddd67429c6179b278d0 Oct 29 22:32:55.017603 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:55.017566 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" event={"ID":"459699a8-f82e-414b-b0df-35d24fdb08a0","Type":"ContainerStarted","Data":"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a"} Oct 29 22:32:55.018084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:55.017632 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" event={"ID":"459699a8-f82e-414b-b0df-35d24fdb08a0","Type":"ContainerStarted","Data":"0d10debc219df80a8cf759dd01cd0053742c81dd1103fddd67429c6179b278d0"} Oct 29 22:32:55.018084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:55.017664 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:32:55.062889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:32:55.062833 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" podStartSLOduration=2.062819955 podStartE2EDuration="2.062819955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:55.061058038 +0000 UTC m=+3033.909534882" watchObservedRunningTime="2026-10-29 22:32:55.062819955 +0000 UTC m=+3033.911296798" Oct 29 22:33:06.029607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:06.029517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:33:16.436571 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:33:16.436529 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:33:46.437292 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:33:46.437248 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:33:54.642470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.642436 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:33:54.647568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.647537 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.657163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.657131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.657352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.657332 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghc7j\" (UniqueName: \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.657505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.657489 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.657617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.657602 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.658015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.657994 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.658185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.658161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.668036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.668004 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:33:54.673578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.673548 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-75ea0c\"/\"openshift-service-ca.crt\"" Oct 29 22:33:54.673578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.673548 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-75ea0c\"/\"kube-root-ca.crt\"" Oct 29 22:33:54.676807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.676786 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-75ea0c\"/\"default-dockercfg-298n5\"" Oct 29 22:33:54.676957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.676840 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-75ea0c\"/\"test-model-75ea0c-kserve-self-signed-certs\"" Oct 29 22:33:54.759898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.759848 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.759898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.759894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc7j\" (UniqueName: \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.759991 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760024 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760271 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760318 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760326 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760566 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.760674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.760600 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.762256 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.762227 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.762621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.762603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.785042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.785010 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc7j\" (UniqueName: \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\") pod \"test-model-75ea0c-kserve-66d5d598cc-zrz8x\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:54.976361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:54.976325 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:33:55.311512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:55.311474 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:33:55.312144 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:33:55.312107 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02414b3a_c3f8_49f9_9cb2_8dd9fe66f15b.slice/crio-52b337ee10c3c1afc8cf665d39f23bbd650e5e40a77a4ead4470811444508c0d WatchSource:0}: Error finding container 52b337ee10c3c1afc8cf665d39f23bbd650e5e40a77a4ead4470811444508c0d: Status 404 returned error can't find the container with id 52b337ee10c3c1afc8cf665d39f23bbd650e5e40a77a4ead4470811444508c0d Oct 29 22:33:55.700625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:55.700514 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" event={"ID":"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b","Type":"ContainerStarted","Data":"342876f93e7b475b81270b538aa177a62a436aa4767b56861ab987eea4a312f4"} Oct 29 22:33:55.700625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:55.700567 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" event={"ID":"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b","Type":"ContainerStarted","Data":"52b337ee10c3c1afc8cf665d39f23bbd650e5e40a77a4ead4470811444508c0d"} Oct 29 22:33:55.744340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:55.744286 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" podStartSLOduration=1.744272598 podStartE2EDuration="1.744272598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:55.743558583 +0000 UTC m=+3094.592035430" watchObservedRunningTime="2026-10-29 22:33:55.744272598 +0000 UTC m=+3094.592749487" Oct 29 22:33:55.978063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:33:55.978025 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:34:04.978461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:04.978371 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:34:04.994333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:04.994303 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:34:05.876133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:05.876090 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:34:09.277023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.276988 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:34:09.286708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.286390 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.294013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.292020 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:34:09.303489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.303463 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-26abe8\"/\"kube-root-ca.crt\"" Oct 29 22:34:09.304891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.304867 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-26abe8\"/\"openshift-service-ca.crt\"" Oct 29 22:34:09.315863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.315838 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-26abe8\"/\"test-model-26abe8-kserve-self-signed-certs\"" Oct 29 22:34:09.316017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.315971 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-26abe8\"/\"default-dockercfg-7qtzw\"" Oct 29 22:34:09.372762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.372723 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.372998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.372786 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.372998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.372931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.372998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.372968 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.373186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.373131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnspv\" (UniqueName: \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.373283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.373262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.475823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.475782 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.475836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.475944 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.475979 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.476127 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnspv\" (UniqueName: \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.476240 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.476326 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.476564 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.476947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.476889 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.478368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.478330 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.478654 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.478633 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.518932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.515223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnspv\" (UniqueName: \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\") pod \"test-model-26abe8-kserve-bb87cf455-lwvt9\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:09.614977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:09.613439 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:10.082277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:10.082246 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:34:10.083341 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:34:10.083313 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772e1d5d_211d_48b3_88fa_108568434c30.slice/crio-212ce0e82e8227625ce7a790f77c91d642ef2c787df31cc636d293600e1c8865 WatchSource:0}: Error finding container 212ce0e82e8227625ce7a790f77c91d642ef2c787df31cc636d293600e1c8865: Status 404 returned error can't find the container with id 212ce0e82e8227625ce7a790f77c91d642ef2c787df31cc636d293600e1c8865 Oct 29 22:34:10.921394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:10.921299 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" event={"ID":"772e1d5d-211d-48b3-88fa-108568434c30","Type":"ContainerStarted","Data":"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d"} Oct 29 22:34:10.921394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:10.921348 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" event={"ID":"772e1d5d-211d-48b3-88fa-108568434c30","Type":"ContainerStarted","Data":"212ce0e82e8227625ce7a790f77c91d642ef2c787df31cc636d293600e1c8865"} Oct 29 22:34:10.966991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:10.966925 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" podStartSLOduration=1.966891527 podStartE2EDuration="1.966891527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.963980832 +0000 UTC m=+3109.812457679" watchObservedRunningTime="2026-10-29 22:34:10.966891527 +0000 UTC m=+3109.815368372" Oct 29 22:34:11.614415 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:11.614376 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:15.966967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.966888 4376 generic.go:356] "Generic (PLEG): container finished" podID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerID="049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88" exitCode=0 Oct 29 22:34:15.967563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.966979 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerDied","Data":"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88"} Oct 29 22:34:15.967650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.967632 4376 scope.go:122] "RemoveContainer" containerID="049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88" Oct 29 22:34:15.970137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.970106 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:15.970242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.970180 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:15.970242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:15.970216 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:16.439670 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:34:16.439579 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:34:16.976741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:16.976693 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerStarted","Data":"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404"} Oct 29 22:34:17.268469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:17.268380 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:17.268469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:17.268439 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:17.316928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:17.309570 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:34:18.060023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:18.059979 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:34:18.267269 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:18.267217 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:18.267506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:18.267291 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:19.266959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:19.266923 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:19.267448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:19.267003 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:19.614469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:19.614382 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:19.631071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:19.631026 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:20.079773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:20.079732 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:34:20.269888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:20.268332 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:20.269888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:20.268429 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:21.271418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:21.271381 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:21.271946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:21.271437 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:22.267129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:22.267089 4376 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 29 22:34:22.267330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:22.267158 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 29 22:34:23.275997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:23.275966 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:34:23.277085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:23.277060 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:34:46.514698 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:34:46.514645 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:34:57.469281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.469244 4376 generic.go:356] "Generic (PLEG): container finished" podID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerID="9a730037e190e90e34d916106368a90f9d170504d2819e0d47e3dadeccfe8025" exitCode=0 Oct 29 22:34:57.469777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.469298 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerDied","Data":"9a730037e190e90e34d916106368a90f9d170504d2819e0d47e3dadeccfe8025"} Oct 29 22:34:57.469777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.469767 4376 scope.go:122] "RemoveContainer" containerID="9a730037e190e90e34d916106368a90f9d170504d2819e0d47e3dadeccfe8025" Oct 29 22:34:57.472238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.472201 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:57.472342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.472279 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:57.472342 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:57.472312 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:34:58.480813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:58.480765 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerStarted","Data":"91bee967ba5b082404651bc29916c6da32ff8768fd1d60d54c803f5e39f42c65"} Oct 29 22:34:59.208894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:59.208858 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:34:59.209170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:59.209145 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:34:59.209268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:34:59.209214 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:00.208482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:00.208449 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:35:00.208989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:00.208504 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:01.209026 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:01.208985 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:35:01.209572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:01.209053 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:02.208475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:02.208432 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:35:02.208670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:02.208506 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:03.209256 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:03.209218 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:35:03.209813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:03.209276 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:04.208352 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:04.208316 4376 patch_prober.go:29] interesting pod/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 22:35:04.208580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:04.208376 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 22:35:05.214837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:05.214801 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:35:05.215387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:05.215361 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:35:05.216029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:35:05.216007 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:35:16.444789 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:35:16.444736 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:35:46.438249 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:35:46.438161 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:36:08.241794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.241754 4376 generic.go:356] "Generic (PLEG): container finished" podID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerID="57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da" exitCode=0 Oct 29 22:36:08.242344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.241813 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerDied","Data":"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da"} Oct 29 22:36:08.242632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.242615 4376 scope.go:122] "RemoveContainer" containerID="57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da" Oct 29 22:36:08.245129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.245098 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:08.245296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.245175 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:08.245296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:08.245210 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:36:09.253217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:09.253178 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerStarted","Data":"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8"} Oct 29 22:36:10.158433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:10.158382 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:10.158850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:10.158818 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:36:10.159035 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:10.158872 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:36:11.162930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:11.162409 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:36:11.162930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:11.162473 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:36:12.156199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:12.155118 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:36:12.156199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:12.155190 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:36:13.158871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:13.158048 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:36:13.158871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:13.158135 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:36:14.152599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:14.152565 4376 patch_prober.go:29] interesting pod/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:36:14.152893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:14.152864 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 29 22:36:15.153894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:15.153865 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:15.154392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:15.154368 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:15.155155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:15.155138 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:16.435921 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:36:16.435872 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:36:31.650882 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.650844 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:36:31.655146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.655123 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.676438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.676403 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:36:31.681451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.681423 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-27b6db\"" Oct 29 22:36:31.681614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.681494 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-27b6db-dockercfg-g7zwb\"" Oct 29 22:36:31.718958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.718889 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.721604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.721537 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rjf\" (UniqueName: \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.865514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.865465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rjf\" (UniqueName: \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.865942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.865915 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.867414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.867391 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.914201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.914111 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rjf\" (UniqueName: \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\") pod \"payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:31.970221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:31.970188 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:32.373011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:32.372982 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:36:32.373177 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:36:32.372981 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439a7a25_91d7_4d82_a5bd_93cb94f10eed.slice/crio-ced9376f073f748ac3d2c3190bbbeb5ce834c1138aacaebc2b791d28f04fde3d WatchSource:0}: Error finding container ced9376f073f748ac3d2c3190bbbeb5ce834c1138aacaebc2b791d28f04fde3d: Status 404 returned error can't find the container with id ced9376f073f748ac3d2c3190bbbeb5ce834c1138aacaebc2b791d28f04fde3d Oct 29 22:36:32.539132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:32.539092 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" event={"ID":"439a7a25-91d7-4d82-a5bd-93cb94f10eed","Type":"ContainerStarted","Data":"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e"} Oct 29 22:36:32.539313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:32.539161 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" event={"ID":"439a7a25-91d7-4d82-a5bd-93cb94f10eed","Type":"ContainerStarted","Data":"ced9376f073f748ac3d2c3190bbbeb5ce834c1138aacaebc2b791d28f04fde3d"} Oct 29 22:36:32.539313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:32.539189 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:32.586587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:32.586535 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" podStartSLOduration=1.5865217820000002 podStartE2EDuration="1.586521782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:32.581844251 +0000 UTC m=+3251.430321096" watchObservedRunningTime="2026-10-29 22:36:32.586521782 +0000 UTC m=+3251.434998627" Oct 29 22:36:34.051305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:34.051269 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:36:34.051978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:34.051525 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" containerID="cri-o://aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8" gracePeriod=30 Oct 29 22:36:39.352422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.352353 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:39.419158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419094 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419169 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419221 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419313 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419338 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419453 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419626 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.419689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.419648 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\") pod \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\" (UID: \"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb\") " Oct 29 22:36:39.421113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.421082 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:39.421442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.421383 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:39.421442 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.421420 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:39.421669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.421514 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:39.421732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.421717 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:39.422769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.422738 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:36:39.423368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.423343 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:39.423480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.423457 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:39.424043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.424022 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp" pod "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" (UID: "1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb"). InnerVolumeSpecName "kube-api-access-m97hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:39.440219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.440188 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:36:39.441852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.441507 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:36:39.441852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.441529 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:36:39.441852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.441576 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:36:39.441852 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.441585 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:36:39.442339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.441862 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:36:39.453246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.453223 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:36:39.453462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.453381 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.471647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.471626 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-219ccc-serving-cert\"" Oct 29 22:36:39.521886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.521849 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.522051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.522183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522167 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.522222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596tr\" (UniqueName: \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.522279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522268 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522282 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522291 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522303 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522347 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97hp\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-kube-api-access-m97hp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522371 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522394 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522408 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.522423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.522420 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.600581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.600544 4376 generic.go:356] "Generic (PLEG): container finished" podID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerID="aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8" exitCode=0 Oct 29 22:36:39.600581 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.600591 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" Oct 29 22:36:39.600840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.600633 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerDied","Data":"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8"} Oct 29 22:36:39.600840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.600689 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr" event={"ID":"1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb","Type":"ContainerDied","Data":"9289aab746bc58a9fd0d367ea157de0a1df48a99e8585d85b21cd5f73f2c9cd4"} Oct 29 22:36:39.600840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.600719 4376 scope.go:122] "RemoveContainer" containerID="aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8" Oct 29 22:36:39.619028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.618992 4376 scope.go:122] "RemoveContainer" containerID="57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da" Oct 29 22:36:39.624475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.624448 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.624584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.624493 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596tr\" (UniqueName: \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.624651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.624636 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.624783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.624764 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.625233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.625209 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.627237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.627213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.627345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.627303 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.632330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.632310 4376 scope.go:122] "RemoveContainer" containerID="aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8" Oct 29 22:36:39.632657 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:36:39.632638 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8\": container with ID starting with aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8 not found: ID does not exist" containerID="aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8" Oct 29 22:36:39.632708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.632666 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8"} err="failed to get container status \"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8\": rpc error: code = NotFound desc = could not find container \"aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8\": container with ID starting with aacad2dab7bedecbd49b720de275e7bf23d8c4ef082b4656d6dcd68c0e2dc7f8 not found: ID does not exist" Oct 29 22:36:39.632708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.632685 4376 scope.go:122] "RemoveContainer" containerID="57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da" Oct 29 22:36:39.633101 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:36:39.633074 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da\": container with ID starting with 57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da not found: ID does not exist" containerID="57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da" Oct 29 22:36:39.633204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.633106 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da"} err="failed to get container status \"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da\": rpc error: code = NotFound desc = could not find container \"57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da\": container with ID starting with 57f81057dd7ba592731404cb209e58174a1c621b7c1a95a377aac8c88f5ca7da not found: ID does not exist" Oct 29 22:36:39.649314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.649295 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596tr\" (UniqueName: \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\") pod \"maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:39.664964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.664942 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:36:39.669465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.669441 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0cb13f-openshift-default-69b44f8b4f-5c8mr"] Oct 29 22:36:39.785373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:39.785336 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:40.097689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.097663 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:36:40.098347 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:36:40.098319 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2522a775_395b_4d22_afc7_a506f4863068.slice/crio-129b0be21bfc8a04661d0ca30931584cdd99552b99a08029938f2867c961ec34 WatchSource:0}: Error finding container 129b0be21bfc8a04661d0ca30931584cdd99552b99a08029938f2867c961ec34: Status 404 returned error can't find the container with id 129b0be21bfc8a04661d0ca30931584cdd99552b99a08029938f2867c961ec34 Oct 29 22:36:40.099878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.099859 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:36:40.610976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.610934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" event={"ID":"2522a775-395b-4d22-afc7-a506f4863068","Type":"ContainerStarted","Data":"6eda12eba9357878e6757e440301a8d0a1ad33fbd365be28fbaa399a41856c05"} Oct 29 22:36:40.611433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.611000 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" event={"ID":"2522a775-395b-4d22-afc7-a506f4863068","Type":"ContainerStarted","Data":"129b0be21bfc8a04661d0ca30931584cdd99552b99a08029938f2867c961ec34"} Oct 29 22:36:40.611433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.611029 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:36:40.656782 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:40.656731 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" podStartSLOduration=154.307487476 podStartE2EDuration="2m34.656716195s" totalImagesPullingTime="349.228719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:40.656684839 +0000 UTC m=+3259.505161683" watchObservedRunningTime="2026-10-29 22:36:40.656716195 +0000 UTC m=+3259.505193038" Oct 29 22:36:41.275732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:41.275698 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" path="/var/lib/kubelet/pods/1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb/volumes" Oct 29 22:36:43.551147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:43.551119 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:36:46.435654 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:36:46.435591 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:36:46.625798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:36:46.625771 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:37:02.128953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:02.128916 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:37:07.129556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.129467 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" podUID="459699a8-f82e-414b-b0df-35d24fdb08a0" containerName="payload-pre-processing" containerID="cri-o://cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a" gracePeriod=25 Oct 29 22:37:07.441289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.441261 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:37:07.482708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.482663 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\" (UniqueName: \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\") pod \"459699a8-f82e-414b-b0df-35d24fdb08a0\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " Oct 29 22:37:07.483011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.482932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\") pod \"459699a8-f82e-414b-b0df-35d24fdb08a0\" (UID: \"459699a8-f82e-414b-b0df-35d24fdb08a0\") " Oct 29 22:37:07.484391 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.484272 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume" pod "459699a8-f82e-414b-b0df-35d24fdb08a0" (UID: "459699a8-f82e-414b-b0df-35d24fdb08a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:07.497085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.492116 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx" pod "459699a8-f82e-414b-b0df-35d24fdb08a0" (UID: "459699a8-f82e-414b-b0df-35d24fdb08a0"). InnerVolumeSpecName "kube-api-access-5f7lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:07.585407 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.585364 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459699a8-f82e-414b-b0df-35d24fdb08a0-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:07.585407 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.585404 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7lx\" (UniqueName: \"kubernetes.io/projected/459699a8-f82e-414b-b0df-35d24fdb08a0-kube-api-access-5f7lx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:07.951986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.951954 4376 generic.go:356] "Generic (PLEG): container finished" podID="459699a8-f82e-414b-b0df-35d24fdb08a0" containerID="cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a" exitCode=0 Oct 29 22:37:07.952171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.952004 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" Oct 29 22:37:07.952171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.952038 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" event={"ID":"459699a8-f82e-414b-b0df-35d24fdb08a0","Type":"ContainerDied","Data":"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a"} Oct 29 22:37:07.952171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.952101 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48" event={"ID":"459699a8-f82e-414b-b0df-35d24fdb08a0","Type":"ContainerDied","Data":"0d10debc219df80a8cf759dd01cd0053742c81dd1103fddd67429c6179b278d0"} Oct 29 22:37:07.952171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.952125 4376 scope.go:122] "RemoveContainer" containerID="cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a" Oct 29 22:37:07.964857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.964830 4376 scope.go:122] "RemoveContainer" containerID="cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a" Oct 29 22:37:07.965141 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:37:07.965122 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a\": container with ID starting with cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a not found: ID does not exist" containerID="cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a" Oct 29 22:37:07.965192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:07.965150 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a"} err="failed to get container status \"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a\": rpc error: code = NotFound desc = could not find container \"cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a\": container with ID starting with cf6390be5f05cd0e6de2b54e29587e45e20564687cf01a8613159665b354fe5a not found: ID does not exist" Oct 29 22:37:08.011094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:08.011049 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:37:08.051926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:08.046949 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-219ccc-55f7fd8cf9-cnc48"] Oct 29 22:37:09.310928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:09.306247 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459699a8-f82e-414b-b0df-35d24fdb08a0" path="/var/lib/kubelet/pods/459699a8-f82e-414b-b0df-35d24fdb08a0/volumes" Oct 29 22:37:16.436116 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:37:16.436070 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:37:17.496082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.496045 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:37:17.496998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.496982 4376 cpu_manager.go:414] "containerMap: removing container" podUID="459699a8-f82e-414b-b0df-35d24fdb08a0" containerName="payload-pre-processing" Oct 29 22:37:17.497050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.496998 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459699a8-f82e-414b-b0df-35d24fdb08a0" containerName="payload-pre-processing" Oct 29 22:37:17.497147 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.497138 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="459699a8-f82e-414b-b0df-35d24fdb08a0" containerName="payload-pre-processing" Oct 29 22:37:17.497186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.497151 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff6ada8-ae62-4a60-9cc5-3d7b82c322eb" containerName="istio-proxy" Oct 29 22:37:17.501000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.500978 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.521178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.521144 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:37:17.525468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.525442 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-563f39-dockercfg-kkt9d\"" Oct 29 22:37:17.526090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.526068 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-563f39\"" Oct 29 22:37:17.526815 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.526792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.527027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.527007 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dbg\" (UniqueName: \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.674197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.653948 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dbg\" (UniqueName: \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.674197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.654501 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.674197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.655618 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.691938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.683378 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dbg\" (UniqueName: \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\") pod \"payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:17.820973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:17.820342 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:18.228530 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:18.228215 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b0d153_a9a9_40fb_9627_02d86d2b68a2.slice/crio-e4f10d79e12cfbcd692df49eab3c91af327e47e81446929b3fff50fada268a53 WatchSource:0}: Error finding container e4f10d79e12cfbcd692df49eab3c91af327e47e81446929b3fff50fada268a53: Status 404 returned error can't find the container with id e4f10d79e12cfbcd692df49eab3c91af327e47e81446929b3fff50fada268a53 Oct 29 22:37:18.233163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:18.232014 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:37:19.050710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:19.050668 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" event={"ID":"04b0d153-a9a9-40fb-9627-02d86d2b68a2","Type":"ContainerStarted","Data":"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed"} Oct 29 22:37:19.051217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:19.050726 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" event={"ID":"04b0d153-a9a9-40fb-9627-02d86d2b68a2","Type":"ContainerStarted","Data":"e4f10d79e12cfbcd692df49eab3c91af327e47e81446929b3fff50fada268a53"} Oct 29 22:37:19.051217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:19.050757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:19.095349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:19.095289 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" podStartSLOduration=2.095274496 podStartE2EDuration="2.095274496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:19.092962171 +0000 UTC m=+3297.941439018" watchObservedRunningTime="2026-10-29 22:37:19.095274496 +0000 UTC m=+3297.943751341" Oct 29 22:37:20.097881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:20.097840 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:37:20.477981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:20.441100 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:37:20.477981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:20.441397 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" containerID="cri-o://bc2bb2b8b66af7226d0fb9ebaa27ae8859619c339f476eca5def407b2cc26231" gracePeriod=30 Oct 29 22:37:21.033281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.033237 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb"] Oct 29 22:37:21.037706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.037670 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.052981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.049939 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtb5\" (UniqueName: \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.052981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.050177 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.052981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.050461 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.060775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.060729 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb"] Oct 29 22:37:21.091636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.091596 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:37:21.091872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.091777 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:37:21.152306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.152263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.152826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.152453 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.152826 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.152676 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtb5\" (UniqueName: \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.153080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.153058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.155079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.155058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.178145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.178102 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtb5\" (UniqueName: \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\") pod \"maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.361978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.361864 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:21.678949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:21.678099 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb"] Oct 29 22:37:21.678949 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:21.678591 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7645db_42a0_499c_8825_d190099f3f98.slice/crio-614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd WatchSource:0}: Error finding container 614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd: Status 404 returned error can't find the container with id 614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd Oct 29 22:37:22.095226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:22.095183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" event={"ID":"0d7645db-42a0-499c-8825-d190099f3f98","Type":"ContainerStarted","Data":"614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd"} Oct 29 22:37:23.105927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:23.105824 4376 generic.go:356] "Generic (PLEG): container finished" podID="0d7645db-42a0-499c-8825-d190099f3f98" containerID="b41fc16f407bd1da84c2de9f5bae2cc5d37b6779975220dc39658fe02ee36535" exitCode=0 Oct 29 22:37:23.106332 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:23.105919 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" event={"ID":"0d7645db-42a0-499c-8825-d190099f3f98","Type":"ContainerDied","Data":"b41fc16f407bd1da84c2de9f5bae2cc5d37b6779975220dc39658fe02ee36535"} Oct 29 22:37:24.342894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.342864 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:24.453445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.453412 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\") pod \"0d7645db-42a0-499c-8825-d190099f3f98\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " Oct 29 22:37:24.453631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.453615 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\" (UniqueName: \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\") pod \"0d7645db-42a0-499c-8825-d190099f3f98\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " Oct 29 22:37:24.453679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.453645 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\") pod \"0d7645db-42a0-499c-8825-d190099f3f98\" (UID: \"0d7645db-42a0-499c-8825-d190099f3f98\") " Oct 29 22:37:24.454099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.454064 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca" pod "0d7645db-42a0-499c-8825-d190099f3f98" (UID: "0d7645db-42a0-499c-8825-d190099f3f98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:24.454327 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.454309 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-openshift-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:24.455674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.455644 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca" pod "0d7645db-42a0-499c-8825-d190099f3f98" (UID: "0d7645db-42a0-499c-8825-d190099f3f98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:24.455799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.455713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5" pod "0d7645db-42a0-499c-8825-d190099f3f98" (UID: "0d7645db-42a0-499c-8825-d190099f3f98"). InnerVolumeSpecName "kube-api-access-rmtb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:24.556572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.556537 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7645db-42a0-499c-8825-d190099f3f98-maas-api-xks-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:24.556572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:24.556565 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmtb5\" (UniqueName: \"kubernetes.io/projected/0d7645db-42a0-499c-8825-d190099f3f98-kube-api-access-rmtb5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:25.123295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:25.123262 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" Oct 29 22:37:25.123468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:25.123294 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb" event={"ID":"0d7645db-42a0-499c-8825-d190099f3f98","Type":"ContainerDied","Data":"614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd"} Oct 29 22:37:25.123468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:25.123347 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614f81307ceac736efa0381b6b241a650d5a1f468b1ed5777249070fd08723fd" Oct 29 22:37:25.740599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:25.740564 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:37:26.012787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.012751 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:37:26.013047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.012993 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 29 22:37:26.149807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.149775 4376 generic.go:356] "Generic (PLEG): container finished" podID="33c5c73a-3d11-4406-b921-d053f710c6da" containerID="bc2bb2b8b66af7226d0fb9ebaa27ae8859619c339f476eca5def407b2cc26231" exitCode=0 Oct 29 22:37:26.150030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.149845 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerDied","Data":"bc2bb2b8b66af7226d0fb9ebaa27ae8859619c339f476eca5def407b2cc26231"} Oct 29 22:37:26.150030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.149942 4376 scope.go:122] "RemoveContainer" containerID="18941031fcdc24f700a1d0abaa522fe5e7435110adeeb982aa4125ab34086b69" Oct 29 22:37:26.201671 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.201632 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv"] Oct 29 22:37:26.202693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.202673 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0d7645db-42a0-499c-8825-d190099f3f98" containerName="revoke-keys" Oct 29 22:37:26.202693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.202694 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7645db-42a0-499c-8825-d190099f3f98" containerName="revoke-keys" Oct 29 22:37:26.202870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.202858 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7645db-42a0-499c-8825-d190099f3f98" containerName="revoke-keys" Oct 29 22:37:26.207433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.207409 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.227618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.227590 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv"] Oct 29 22:37:26.230110 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.230086 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:37:26.230307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.230288 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:37:26.408303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.407161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.408303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.407494 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.408303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.407571 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnzl\" (UniqueName: \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.436740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.436711 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:37:26.509745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.508950 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:37:26.510496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.510468 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.511020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.511001 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.511631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.511613 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.512144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.512125 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.512548 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.512527 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.513099 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.513066 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.513788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.513769 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.514451 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.510884 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.514641 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.512022 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.515598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.512474 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.515739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.513535 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:26.515855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.514158 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.515978 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.511478 4376 cpu_manager.go:414] "containerMap: removing container" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:26.516082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.516068 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:26.516602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.516581 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:26.516755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.516741 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:26.517793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.517763 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:26.517959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.517944 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.518422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.518405 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\") pod \"33c5c73a-3d11-4406-b921-d053f710c6da\" (UID: \"33c5c73a-3d11-4406-b921-d053f710c6da\") " Oct 29 22:37:26.523124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.523088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "kube-api-access-k7krs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:26.523595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.523556 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.536168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.536129 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnzl\" (UniqueName: \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.540893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.540872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.541692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.541668 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542020 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542159 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542252 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542347 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c5c73a-3d11-4406-b921-d053f710c6da-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542473 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.542579 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7krs\" (UniqueName: \"kubernetes.io/projected/33c5c73a-3d11-4406-b921-d053f710c6da-kube-api-access-k7krs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.542710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.524338 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.546184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.546156 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.546916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.546577 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:37:26.546916 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.546658 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.555586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.555551 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo" pod "33c5c73a-3d11-4406-b921-d053f710c6da" (UID: "33c5c73a-3d11-4406-b921-d053f710c6da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:37:26.556031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.556013 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.573748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.573719 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnzl\" (UniqueName: \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\") pod \"maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.644176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644082 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644229 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644302 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644423 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644540 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644592 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn9z\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644758 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.644944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.644925 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.645065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.645053 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c5c73a-3d11-4406-b921-d053f710c6da-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.645157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.645070 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c5c73a-3d11-4406-b921-d053f710c6da-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747187 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747301 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn9z\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747395 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747424 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.747739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747713 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747819 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747861 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747873 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.747940 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.748177 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.748570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.748553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.749870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.749846 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.750328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.750308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.772506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.772482 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.772728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.772699 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn9z\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\") pod \"e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:26.825929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.825874 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:26.903255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:26.903223 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:27.150755 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:27.150685 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60deb82e_d180_4721_bd13_183550d6997c.slice/crio-59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40 WatchSource:0}: Error finding container 59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40: Status 404 returned error can't find the container with id 59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40 Oct 29 22:37:27.151820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.151788 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv"] Oct 29 22:37:27.159942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.159895 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" Oct 29 22:37:27.160029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.159939 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47" event={"ID":"33c5c73a-3d11-4406-b921-d053f710c6da","Type":"ContainerDied","Data":"daa79dd2a92bb9f10c2e1420905eabc0997f4045e84f6dd737db371dcaad5d76"} Oct 29 22:37:27.160029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.160008 4376 scope.go:122] "RemoveContainer" containerID="bc2bb2b8b66af7226d0fb9ebaa27ae8859619c339f476eca5def407b2cc26231" Oct 29 22:37:27.220711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.220680 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:37:27.229423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.229397 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-96g47"] Oct 29 22:37:27.236751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.236721 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:37:27.275625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.275588 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" path="/var/lib/kubelet/pods/33c5c73a-3d11-4406-b921-d053f710c6da/volumes" Oct 29 22:37:27.337216 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:27.337177 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411dd6f0_fe08_41b2_982d_d2cdfe258fad.slice/crio-0d8d1822282cdd13c42eb24b6ddcf3809478689ae9d50282c59ba1b3abd18055 WatchSource:0}: Error finding container 0d8d1822282cdd13c42eb24b6ddcf3809478689ae9d50282c59ba1b3abd18055: Status 404 returned error can't find the container with id 0d8d1822282cdd13c42eb24b6ddcf3809478689ae9d50282c59ba1b3abd18055 Oct 29 22:37:27.341021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.340978 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:27.341116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.341075 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:27.341163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:27.341118 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:28.217488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.217446 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" event={"ID":"60deb82e-d180-4721-bd13-183550d6997c","Type":"ContainerStarted","Data":"1b13a98f337ca69b9d592a6886f1d4f1ad29bccbacc48828aba54e8b8b52abcf"} Oct 29 22:37:28.218384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.217561 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" event={"ID":"60deb82e-d180-4721-bd13-183550d6997c","Type":"ContainerStarted","Data":"59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40"} Oct 29 22:37:28.220832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.220804 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" event={"ID":"411dd6f0-fe08-41b2-982d-d2cdfe258fad","Type":"ContainerStarted","Data":"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914"} Oct 29 22:37:28.221003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.220848 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" event={"ID":"411dd6f0-fe08-41b2-982d-d2cdfe258fad","Type":"ContainerStarted","Data":"0d8d1822282cdd13c42eb24b6ddcf3809478689ae9d50282c59ba1b3abd18055"} Oct 29 22:37:28.267394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.267335 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" podStartSLOduration=2.267320432 podStartE2EDuration="2.267320432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:28.261686999 +0000 UTC m=+3307.110163882" watchObservedRunningTime="2026-10-29 22:37:28.267320432 +0000 UTC m=+3307.115797277" Oct 29 22:37:28.314129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.314070 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podStartSLOduration=181.314056149 podStartE2EDuration="3m1.314056149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:28.310590787 +0000 UTC m=+3307.159067633" watchObservedRunningTime="2026-10-29 22:37:28.314056149 +0000 UTC m=+3307.162533035" Oct 29 22:37:28.904763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.904732 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:28.906734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.906708 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:28.906872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:28.906755 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:29.230810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:29.230778 4376 generic.go:356] "Generic (PLEG): container finished" podID="60deb82e-d180-4721-bd13-183550d6997c" containerID="1b13a98f337ca69b9d592a6886f1d4f1ad29bccbacc48828aba54e8b8b52abcf" exitCode=0 Oct 29 22:37:29.231306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:29.230873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" event={"ID":"60deb82e-d180-4721-bd13-183550d6997c","Type":"ContainerDied","Data":"1b13a98f337ca69b9d592a6886f1d4f1ad29bccbacc48828aba54e8b8b52abcf"} Oct 29 22:37:29.904386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:29.904347 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:29.904605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:29.904409 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:30.084179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.084143 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:37:30.514824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.514796 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:30.602868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.602771 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\") pod \"60deb82e-d180-4721-bd13-183550d6997c\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " Oct 29 22:37:30.603069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.602957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\" (UniqueName: \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\") pod \"60deb82e-d180-4721-bd13-183550d6997c\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " Oct 29 22:37:30.603184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.603159 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca" pod "60deb82e-d180-4721-bd13-183550d6997c" (UID: "60deb82e-d180-4721-bd13-183550d6997c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:30.603254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.603206 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\") pod \"60deb82e-d180-4721-bd13-183550d6997c\" (UID: \"60deb82e-d180-4721-bd13-183550d6997c\") " Oct 29 22:37:30.603888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.603863 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60deb82e-d180-4721-bd13-183550d6997c-maas-api-openshift-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:30.605243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.605212 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca" pod "60deb82e-d180-4721-bd13-183550d6997c" (UID: "60deb82e-d180-4721-bd13-183550d6997c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:30.605389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.605364 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl" pod "60deb82e-d180-4721-bd13-183550d6997c" (UID: "60deb82e-d180-4721-bd13-183550d6997c"). InnerVolumeSpecName "kube-api-access-rwnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:30.705717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.705681 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60deb82e-d180-4721-bd13-183550d6997c-maas-api-xks-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:30.705717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.705713 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnzl\" (UniqueName: \"kubernetes.io/projected/60deb82e-d180-4721-bd13-183550d6997c-kube-api-access-rwnzl\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:30.905333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.905251 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:30.905333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:30.905305 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:31.285194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:31.285155 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" event={"ID":"60deb82e-d180-4721-bd13-183550d6997c","Type":"ContainerDied","Data":"59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40"} Oct 29 22:37:31.285194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:31.285177 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv" Oct 29 22:37:31.285491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:31.285216 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b680d54cb4025c4316d6422efb51022fb129c09157ca8445bc17eeaf86af40" Oct 29 22:37:31.904439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:31.904400 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:31.904829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:31.904450 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:32.904477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:32.904440 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:32.904935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:32.904500 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:33.905240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:33.905200 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 29 22:37:33.905611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:33.905257 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 29 22:37:34.910062 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:34.910030 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:34.910524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:34.910446 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:34.910968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:34.910950 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:37:35.121418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.121373 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" podUID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" containerName="main" containerID="cri-o://342876f93e7b475b81270b538aa177a62a436aa4767b56861ab987eea4a312f4" gracePeriod=45 Oct 29 22:37:35.393454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.393407 4376 generic.go:356] "Generic (PLEG): container finished" podID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" containerID="342876f93e7b475b81270b538aa177a62a436aa4767b56861ab987eea4a312f4" exitCode=0 Oct 29 22:37:35.394058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.394018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" event={"ID":"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b","Type":"ContainerDied","Data":"342876f93e7b475b81270b538aa177a62a436aa4767b56861ab987eea4a312f4"} Oct 29 22:37:35.497242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.497175 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:37:35.580298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580268 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580359 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\" (UniqueName: \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580387 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580555 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\" (UniqueName: \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580560 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:35.580643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580586 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580629 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\") pod \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\" (UID: \"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b\") " Oct 29 22:37:35.580800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.580725 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:35.581190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.581113 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:35.581598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.581572 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:35.581697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.581603 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:35.581697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.581616 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:35.583235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.583208 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:35.583455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.583431 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "kube-api-access-ghc7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:35.584223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.584198 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm" pod "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" (UID: "02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:35.683278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.683239 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghc7j\" (UniqueName: \"kubernetes.io/projected/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-kube-api-access-ghc7j\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:35.683278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.683270 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:35.683278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:35.683280 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:36.403331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:36.403301 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" Oct 29 22:37:36.403840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:36.403339 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x" event={"ID":"02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b","Type":"ContainerDied","Data":"52b337ee10c3c1afc8cf665d39f23bbd650e5e40a77a4ead4470811444508c0d"} Oct 29 22:37:36.403840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:36.403381 4376 scope.go:122] "RemoveContainer" containerID="342876f93e7b475b81270b538aa177a62a436aa4767b56861ab987eea4a312f4" Oct 29 22:37:36.475530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:36.475493 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:37:36.480381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:36.480356 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-75ea0c/test-model-75ea0c-kserve-66d5d598cc-zrz8x"] Oct 29 22:37:37.276409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:37.276373 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" path="/var/lib/kubelet/pods/02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b/volumes" Oct 29 22:37:40.766673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:40.766623 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" podUID="772e1d5d-211d-48b3-88fa-108568434c30" containerName="main" containerID="cri-o://5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d" gracePeriod=45 Oct 29 22:37:41.062320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.062292 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:37:41.136063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136024 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136074 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136217 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136331 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\" (UniqueName: \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136442 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:41.136599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136487 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136520 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\") pod \"772e1d5d-211d-48b3-88fa-108568434c30\" (UID: \"772e1d5d-211d-48b3-88fa-108568434c30\") " Oct 29 22:37:41.136711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136485 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:41.136711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.136692 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:41.137335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.137310 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.137455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.137338 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.137455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.137351 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.139080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.139041 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:41.139194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.139168 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "kube-api-access-qnspv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:41.139278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.139183 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm" pod "772e1d5d-211d-48b3-88fa-108568434c30" (UID: "772e1d5d-211d-48b3-88fa-108568434c30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:41.238660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.238627 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnspv\" (UniqueName: \"kubernetes.io/projected/772e1d5d-211d-48b3-88fa-108568434c30-kube-api-access-qnspv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.238660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.238656 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1d5d-211d-48b3-88fa-108568434c30-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.238660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.238668 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1d5d-211d-48b3-88fa-108568434c30-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:41.558158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.558063 4376 generic.go:356] "Generic (PLEG): container finished" podID="772e1d5d-211d-48b3-88fa-108568434c30" containerID="5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d" exitCode=0 Oct 29 22:37:41.558158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.558123 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" Oct 29 22:37:41.558411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.558151 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" event={"ID":"772e1d5d-211d-48b3-88fa-108568434c30","Type":"ContainerDied","Data":"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d"} Oct 29 22:37:41.558411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.558199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9" event={"ID":"772e1d5d-211d-48b3-88fa-108568434c30","Type":"ContainerDied","Data":"212ce0e82e8227625ce7a790f77c91d642ef2c787df31cc636d293600e1c8865"} Oct 29 22:37:41.558411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.558220 4376 scope.go:122] "RemoveContainer" containerID="5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d" Oct 29 22:37:41.570075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.570056 4376 scope.go:122] "RemoveContainer" containerID="5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d" Oct 29 22:37:41.570346 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:37:41.570325 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d\": container with ID starting with 5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d not found: ID does not exist" containerID="5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d" Oct 29 22:37:41.570455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.570353 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d"} err="failed to get container status \"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d\": rpc error: code = NotFound desc = could not find container \"5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d\": container with ID starting with 5e8daf1339590c3be19b300106b7700cb1285af8a7214e918944f9ecb3da7a1d not found: ID does not exist" Oct 29 22:37:41.616598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.616567 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:37:41.629053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:41.629021 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-26abe8/test-model-26abe8-kserve-bb87cf455-lwvt9"] Oct 29 22:37:43.289722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:43.289678 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772e1d5d-211d-48b3-88fa-108568434c30" path="/var/lib/kubelet/pods/772e1d5d-211d-48b3-88fa-108568434c30/volumes" Oct 29 22:37:44.457457 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.457422 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:37:44.458268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.458241 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" podUID="2522a775-395b-4d22-afc7-a506f4863068" containerName="maas-api" containerID="cri-o://6eda12eba9357878e6757e440301a8d0a1ad33fbd365be28fbaa399a41856c05" gracePeriod=30 Oct 29 22:37:44.584381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.584343 4376 generic.go:356] "Generic (PLEG): container finished" podID="2522a775-395b-4d22-afc7-a506f4863068" containerID="6eda12eba9357878e6757e440301a8d0a1ad33fbd365be28fbaa399a41856c05" exitCode=0 Oct 29 22:37:44.584559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.584412 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" event={"ID":"2522a775-395b-4d22-afc7-a506f4863068","Type":"ContainerDied","Data":"6eda12eba9357878e6757e440301a8d0a1ad33fbd365be28fbaa399a41856c05"} Oct 29 22:37:44.887340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.887308 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:37:44.950724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.950689 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\" (UniqueName: \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\") pod \"2522a775-395b-4d22-afc7-a506f4863068\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " Oct 29 22:37:44.950961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.950773 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\") pod \"2522a775-395b-4d22-afc7-a506f4863068\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " Oct 29 22:37:44.950961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.950927 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\") pod \"2522a775-395b-4d22-afc7-a506f4863068\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " Oct 29 22:37:44.951416 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.951102 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\") pod \"2522a775-395b-4d22-afc7-a506f4863068\" (UID: \"2522a775-395b-4d22-afc7-a506f4863068\") " Oct 29 22:37:44.951633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.951605 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca" pod "2522a775-395b-4d22-afc7-a506f4863068" (UID: "2522a775-395b-4d22-afc7-a506f4863068"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:44.952313 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.952283 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2522a775-395b-4d22-afc7-a506f4863068-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:44.954318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.954293 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls" pod "2522a775-395b-4d22-afc7-a506f4863068" (UID: "2522a775-395b-4d22-afc7-a506f4863068"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:44.954658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.954532 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr" pod "2522a775-395b-4d22-afc7-a506f4863068" (UID: "2522a775-395b-4d22-afc7-a506f4863068"). InnerVolumeSpecName "kube-api-access-596tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:44.956208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:44.956181 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs" pod "2522a775-395b-4d22-afc7-a506f4863068" (UID: "2522a775-395b-4d22-afc7-a506f4863068"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:45.056462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.056428 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:45.056462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.056460 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596tr\" (UniqueName: \"kubernetes.io/projected/2522a775-395b-4d22-afc7-a506f4863068-kube-api-access-596tr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:45.056462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.056470 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2522a775-395b-4d22-afc7-a506f4863068-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:45.668989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.668751 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" event={"ID":"2522a775-395b-4d22-afc7-a506f4863068","Type":"ContainerDied","Data":"129b0be21bfc8a04661d0ca30931584cdd99552b99a08029938f2867c961ec34"} Oct 29 22:37:45.668989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.668838 4376 scope.go:122] "RemoveContainer" containerID="6eda12eba9357878e6757e440301a8d0a1ad33fbd365be28fbaa399a41856c05" Oct 29 22:37:45.669620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.669014 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6" Oct 29 22:37:45.783375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.783311 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:37:45.829932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:45.827855 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219ccc-866b9898cc-flwc6"] Oct 29 22:37:46.437255 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:37:46.437183 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:37:47.288473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:47.288438 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2522a775-395b-4d22-afc7-a506f4863068" path="/var/lib/kubelet/pods/2522a775-395b-4d22-afc7-a506f4863068/volumes" Oct 29 22:37:52.745919 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.745880 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:37:52.746940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.746890 4376 cpu_manager.go:414] "containerMap: removing container" podUID="772e1d5d-211d-48b3-88fa-108568434c30" containerName="main" Oct 29 22:37:52.746940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.746940 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772e1d5d-211d-48b3-88fa-108568434c30" containerName="main" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.746987 4376 cpu_manager.go:414] "containerMap: removing container" podUID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" containerName="main" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.746993 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" containerName="main" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747033 4376 cpu_manager.go:414] "containerMap: removing container" podUID="60deb82e-d180-4721-bd13-183550d6997c" containerName="revoke-keys" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747038 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60deb82e-d180-4721-bd13-183550d6997c" containerName="revoke-keys" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747047 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2522a775-395b-4d22-afc7-a506f4863068" containerName="maas-api" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747052 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2522a775-395b-4d22-afc7-a506f4863068" containerName="maas-api" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747060 4376 cpu_manager.go:414] "containerMap: removing container" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:52.747224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747065 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5c73a-3d11-4406-b921-d053f710c6da" containerName="istio-proxy" Oct 29 22:37:52.747687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747259 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="772e1d5d-211d-48b3-88fa-108568434c30" containerName="main" Oct 29 22:37:52.747687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747281 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="02414b3a-c3f8-49f9-9cb2-8dd9fe66f15b" containerName="main" Oct 29 22:37:52.747687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747301 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="60deb82e-d180-4721-bd13-183550d6997c" containerName="revoke-keys" Oct 29 22:37:52.747687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.747310 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2522a775-395b-4d22-afc7-a506f4863068" containerName="maas-api" Oct 29 22:37:52.753186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.753167 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.772188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.772161 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:37:52.795158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.795121 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52429a-openshift-default-dockercfg-rbvt2\"" Oct 29 22:37:52.859490 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859455 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859499 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg89l\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859587 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859643 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859816 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859858 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.859890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859879 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.860012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.859997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.860137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.860120 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.961867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.961830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.961874 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg89l\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.961950 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962009 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962174 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962150 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962191 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962223 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962281 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962393 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962433 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962459 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962505 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.962732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.962713 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.963075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.963058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.964677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.964654 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.964816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.964797 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.987183 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.987157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:52.988149 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:52.988129 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg89l\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:53.071876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.071808 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:53.332428 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.332347 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:37:53.332610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.332552 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" containerID="cri-o://91bee967ba5b082404651bc29916c6da32ff8768fd1d60d54c803f5e39f42c65" gracePeriod=30 Oct 29 22:37:53.416365 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:53.416332 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0b6e60_1d0c_4bae_b608_078ef90ec583.slice/crio-59ed01f5e3601152bb54b377853d5faa6064c645e717c333b5a5785a7eabae82 WatchSource:0}: Error finding container 59ed01f5e3601152bb54b377853d5faa6064c645e717c333b5a5785a7eabae82: Status 404 returned error can't find the container with id 59ed01f5e3601152bb54b377853d5faa6064c645e717c333b5a5785a7eabae82 Oct 29 22:37:53.417184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.417099 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:37:53.420558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.420521 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:53.420666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.420591 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:53.420666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.420630 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:37:53.760616 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.760577 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" event={"ID":"1e0b6e60-1d0c-4bae-b608-078ef90ec583","Type":"ContainerStarted","Data":"1a000b22f4dbe6b046a5d5668913c2d42e0e0d6e261062d51da86f2d0d1054df"} Oct 29 22:37:53.761126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.760645 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" event={"ID":"1e0b6e60-1d0c-4bae-b608-078ef90ec583","Type":"ContainerStarted","Data":"59ed01f5e3601152bb54b377853d5faa6064c645e717c333b5a5785a7eabae82"} Oct 29 22:37:53.817555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:53.817501 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podStartSLOduration=1.817486141 podStartE2EDuration="1.817486141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:53.813641154 +0000 UTC m=+3332.662117995" watchObservedRunningTime="2026-10-29 22:37:53.817486141 +0000 UTC m=+3332.665962986" Oct 29 22:37:54.072170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:54.072129 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:37:54.073336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:54.073304 4376 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 29 22:37:54.073472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:54.073356 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 29 22:37:55.072731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.072696 4376 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 29 22:37:55.073145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.072747 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 29 22:37:55.373757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.373679 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr"] Oct 29 22:37:55.379113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.379083 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.399866 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.399835 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr"] Oct 29 22:37:55.404499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.404471 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:37:55.404628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.404577 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:37:55.517291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.517257 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p67s\" (UniqueName: \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.517465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.517320 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.517465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.517405 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.618973 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.618939 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.619253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.619235 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p67s\" (UniqueName: \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.619301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.619285 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.619958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.619928 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.621398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.621369 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.659031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.657656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p67s\" (UniqueName: \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\") pod \"maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:55.697980 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:55.697944 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:37:56.062563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.062525 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr"] Oct 29 22:37:56.063241 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:37:56.063207 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef2ee4d_cb06_4386_8909_c6f467f5d35d.slice/crio-f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a WatchSource:0}: Error finding container f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a: Status 404 returned error can't find the container with id f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a Oct 29 22:37:56.073271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.073245 4376 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 29 22:37:56.073665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.073291 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 29 22:37:56.788022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.787986 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" event={"ID":"aef2ee4d-cb06-4386-8909-c6f467f5d35d","Type":"ContainerStarted","Data":"015d3c21332e889968ee17a3dc779f73328a9b3dabe421548d13809e80a28234"} Oct 29 22:37:56.788228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.788074 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" event={"ID":"aef2ee4d-cb06-4386-8909-c6f467f5d35d","Type":"ContainerStarted","Data":"f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a"} Oct 29 22:37:56.863791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:56.863665 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" podStartSLOduration=1.863644808 podStartE2EDuration="1.863644808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:56.849622835 +0000 UTC m=+3335.698099682" watchObservedRunningTime="2026-10-29 22:37:56.863644808 +0000 UTC m=+3335.712121655" Oct 29 22:37:57.072982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:57.072945 4376 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 29 22:37:57.073182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:57.073008 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 29 22:37:58.072182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.072147 4376 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 29 22:37:58.072574 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.072211 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 29 22:37:58.231957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.229638 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:37:58.231957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.229874 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" containerID="cri-o://1a000b22f4dbe6b046a5d5668913c2d42e0e0d6e261062d51da86f2d0d1054df" gracePeriod=30 Oct 29 22:37:58.814132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.813507 4376 generic.go:356] "Generic (PLEG): container finished" podID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerID="91bee967ba5b082404651bc29916c6da32ff8768fd1d60d54c803f5e39f42c65" exitCode=0 Oct 29 22:37:58.814132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.813860 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerDied","Data":"91bee967ba5b082404651bc29916c6da32ff8768fd1d60d54c803f5e39f42c65"} Oct 29 22:37:58.814132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.813962 4376 scope.go:122] "RemoveContainer" containerID="9a730037e190e90e34d916106368a90f9d170504d2819e0d47e3dadeccfe8025" Oct 29 22:37:58.818849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.818807 4376 generic.go:356] "Generic (PLEG): container finished" podID="aef2ee4d-cb06-4386-8909-c6f467f5d35d" containerID="015d3c21332e889968ee17a3dc779f73328a9b3dabe421548d13809e80a28234" exitCode=0 Oct 29 22:37:58.819071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.819045 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" event={"ID":"aef2ee4d-cb06-4386-8909-c6f467f5d35d","Type":"ContainerDied","Data":"015d3c21332e889968ee17a3dc779f73328a9b3dabe421548d13809e80a28234"} Oct 29 22:37:58.977420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:58.977385 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:37:59.066748 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.066633 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.067439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067360 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.067439 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067420 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.067697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067677 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.067773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067742 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.067870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067856 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.068001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067985 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.068071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.068033 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.068170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.068157 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\") pod \"a6add151-f8c2-433f-9917-b751ae22cbd1\" (UID: \"a6add151-f8c2-433f-9917-b751ae22cbd1\") " Oct 29 22:37:59.069432 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.067015 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:59.069556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.069467 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:59.069556 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.069223 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:59.070096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.070018 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.070096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.070049 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.070096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.070065 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.070660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.070622 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:59.071159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.071134 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "kube-api-access-5fpxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:59.071298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.071152 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:59.072471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.072443 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:59.072950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.072924 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:59.073059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.073026 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo" pod "a6add151-f8c2-433f-9917-b751ae22cbd1" (UID: "a6add151-f8c2-433f-9917-b751ae22cbd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:37:59.172097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172053 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.172097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172095 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.172097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172104 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.172097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172113 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6add151-f8c2-433f-9917-b751ae22cbd1-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.172369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172123 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fpxt\" (UniqueName: \"kubernetes.io/projected/a6add151-f8c2-433f-9917-b751ae22cbd1-kube-api-access-5fpxt\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.172369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.172132 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6add151-f8c2-433f-9917-b751ae22cbd1-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:59.870038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.870000 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" Oct 29 22:37:59.870038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.870022 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb" event={"ID":"a6add151-f8c2-433f-9917-b751ae22cbd1","Type":"ContainerDied","Data":"9c1d37792f364d6588552c02534c3631316952633727b56aebf5dae0885c1a1f"} Oct 29 22:37:59.870294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.870106 4376 scope.go:122] "RemoveContainer" containerID="91bee967ba5b082404651bc29916c6da32ff8768fd1d60d54c803f5e39f42c65" Oct 29 22:37:59.931527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.931501 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:37:59.969822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:37:59.966448 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-219ccc-openshift-default-58d9fff665-bknlb"] Oct 29 22:38:00.111425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.111398 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:38:00.292375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.292339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\") pod \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " Oct 29 22:38:00.292705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.292676 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\") pod \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " Oct 29 22:38:00.292864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.292707 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca" pod "aef2ee4d-cb06-4386-8909-c6f467f5d35d" (UID: "aef2ee4d-cb06-4386-8909-c6f467f5d35d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:00.292864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.292727 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\" (UniqueName: \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\") pod \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\" (UID: \"aef2ee4d-cb06-4386-8909-c6f467f5d35d\") " Oct 29 22:38:00.293368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.293341 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-openshift-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.295156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.295133 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca" pod "aef2ee4d-cb06-4386-8909-c6f467f5d35d" (UID: "aef2ee4d-cb06-4386-8909-c6f467f5d35d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:00.295281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.295260 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s" pod "aef2ee4d-cb06-4386-8909-c6f467f5d35d" (UID: "aef2ee4d-cb06-4386-8909-c6f467f5d35d"). InnerVolumeSpecName "kube-api-access-7p67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:00.400734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.400704 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef2ee4d-cb06-4386-8909-c6f467f5d35d-maas-api-xks-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.400734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.400736 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p67s\" (UniqueName: \"kubernetes.io/projected/aef2ee4d-cb06-4386-8909-c6f467f5d35d-kube-api-access-7p67s\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.883018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.881820 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" Oct 29 22:38:00.883018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.881846 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr" event={"ID":"aef2ee4d-cb06-4386-8909-c6f467f5d35d","Type":"ContainerDied","Data":"f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a"} Oct 29 22:38:00.883018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:00.881899 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7677e9af8b43ba78c6ebcb57dd8eb471bd66197ae67a0b9bde604cdc2217d1a" Oct 29 22:38:01.279453 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:01.279422 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" path="/var/lib/kubelet/pods/a6add151-f8c2-433f-9917-b751ae22cbd1/volumes" Oct 29 22:38:05.936216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:05.936183 4376 generic.go:356] "Generic (PLEG): container finished" podID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerID="1a000b22f4dbe6b046a5d5668913c2d42e0e0d6e261062d51da86f2d0d1054df" exitCode=0 Oct 29 22:38:05.936622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:05.936263 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" event={"ID":"1e0b6e60-1d0c-4bae-b608-078ef90ec583","Type":"ContainerDied","Data":"1a000b22f4dbe6b046a5d5668913c2d42e0e0d6e261062d51da86f2d0d1054df"} Oct 29 22:38:06.102525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.102463 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:38:06.217719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.217686 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.217883 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.217744 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218057 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218103 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218141 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218173 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218204 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218420 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\") pod \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\" (UID: \"1e0b6e60-1d0c-4bae-b608-078ef90ec583\") " Oct 29 22:38:06.218766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218733 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:06.218766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218743 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:06.219017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.218993 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:06.219103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:06.219103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219085 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:06.219434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219408 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.219537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219439 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.219537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219456 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.219537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219472 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.219537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.219487 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.220335 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.220299 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:06.220551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.220531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:06.220856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.220837 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "kube-api-access-jg89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:06.220957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.220868 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy" pod "1e0b6e60-1d0c-4bae-b608-078ef90ec583" (UID: "1e0b6e60-1d0c-4bae-b608-078ef90ec583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:06.321287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.321250 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.321287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.321283 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.321511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.321296 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg89l\" (UniqueName: \"kubernetes.io/projected/1e0b6e60-1d0c-4bae-b608-078ef90ec583-kube-api-access-jg89l\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.321511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.321309 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b6e60-1d0c-4bae-b608-078ef90ec583-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:06.946681 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.946653 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" Oct 29 22:38:06.947175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.946643 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx" event={"ID":"1e0b6e60-1d0c-4bae-b608-078ef90ec583","Type":"ContainerDied","Data":"59ed01f5e3601152bb54b377853d5faa6064c645e717c333b5a5785a7eabae82"} Oct 29 22:38:06.947175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:06.946790 4376 scope.go:122] "RemoveContainer" containerID="1a000b22f4dbe6b046a5d5668913c2d42e0e0d6e261062d51da86f2d0d1054df" Oct 29 22:38:07.026426 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:07.026391 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:38:07.038372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:07.038347 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-5tqrx"] Oct 29 22:38:07.275576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:07.275544 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" path="/var/lib/kubelet/pods/1e0b6e60-1d0c-4bae-b608-078ef90ec583/volumes" Oct 29 22:38:10.796077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:10.796042 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:38:15.342591 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:15.342549 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv"] Oct 29 22:38:15.353709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:15.353672 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75ea0c-fmnwv"] Oct 29 22:38:15.796337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:15.796288 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" containerID="cri-o://75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0" gracePeriod=25 Oct 29 22:38:16.437037 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:38:16.436996 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:38:17.275014 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:17.274977 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60deb82e-d180-4721-bd13-183550d6997c" path="/var/lib/kubelet/pods/60deb82e-d180-4721-bd13-183550d6997c/volumes" Oct 29 22:38:19.463864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:19.463818 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:23.952096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:23.952060 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:38:23.952659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:23.952302 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" containerID="cri-o://0eb410e3ccbc8e4f2189b81dab21eee741e9979829f570d2247bf9c25f46fcf1" gracePeriod=30 Oct 29 22:38:29.283857 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.276229 4376 generic.go:356] "Generic (PLEG): container finished" podID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerID="0eb410e3ccbc8e4f2189b81dab21eee741e9979829f570d2247bf9c25f46fcf1" exitCode=0 Oct 29 22:38:29.290064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.290014 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerDied","Data":"0eb410e3ccbc8e4f2189b81dab21eee741e9979829f570d2247bf9c25f46fcf1"} Oct 29 22:38:29.290351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.290330 4376 scope.go:122] "RemoveContainer" containerID="d290e41260584899e16f00ca6b124f928cf6bf75a3d29219b7c38b2f52cbf1db" Oct 29 22:38:29.455388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.454963 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:29.491760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.491723 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:38:29.577835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.577755 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.577835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.577826 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.578093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.577861 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.578093 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.577964 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.578214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.578097 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.579178 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579148 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:29.579328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579237 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:29.579559 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.579637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579577 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.579798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.579870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579834 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\") pod \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\" (UID: \"12b69987-5bf8-4949-82b0-7ca1f4e0eac3\") " Oct 29 22:38:29.579979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.579955 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:29.580197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580179 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:29.580629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580606 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:29.580720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580640 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.580720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580661 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.580720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580674 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.580720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.580689 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.582880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.582854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:29.583343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.583321 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:29.585218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.585193 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:29.585949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.585405 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp" pod "12b69987-5bf8-4949-82b0-7ca1f4e0eac3" (UID: "12b69987-5bf8-4949-82b0-7ca1f4e0eac3"). InnerVolumeSpecName "kube-api-access-sqfqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:29.682072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.682036 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.682072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.682065 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.682072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.682075 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfqp\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-kube-api-access-sqfqp\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.682318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.682085 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.682318 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:29.682095 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b69987-5bf8-4949-82b0-7ca1f4e0eac3-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.233933 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.233883 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr"] Oct 29 22:38:30.245090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.245059 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0412c3-r9ftr"] Oct 29 22:38:30.286740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.286700 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" Oct 29 22:38:30.286740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.286709 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd" event={"ID":"12b69987-5bf8-4949-82b0-7ca1f4e0eac3","Type":"ContainerDied","Data":"1807aaff8703e9bfadbb08ca49bd87d7fb3572d55d52099d7da4f7499bb4f144"} Oct 29 22:38:30.287279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.286773 4376 scope.go:122] "RemoveContainer" containerID="0eb410e3ccbc8e4f2189b81dab21eee741e9979829f570d2247bf9c25f46fcf1" Oct 29 22:38:30.303591 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.303565 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:38:30.304542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304522 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304542 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304569 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aef2ee4d-cb06-4386-8909-c6f467f5d35d" containerName="revoke-keys" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304575 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef2ee4d-cb06-4386-8909-c6f467f5d35d" containerName="revoke-keys" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304596 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304601 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304621 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.304629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304629 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304662 4376 cpu_manager.go:414] "containerMap: removing container" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304667 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304674 4376 cpu_manager.go:414] "containerMap: removing container" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304679 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304820 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304831 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304839 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0b6e60-1d0c-4bae-b608-078ef90ec583" containerName="istio-proxy" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304849 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef2ee4d-cb06-4386-8909-c6f467f5d35d" containerName="revoke-keys" Oct 29 22:38:30.304856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304858 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.305158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.304865 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6add151-f8c2-433f-9917-b751ae22cbd1" containerName="istio-proxy" Oct 29 22:38:30.310675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.310654 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.330709 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.330682 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:38:30.331042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.331019 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-563f39-serving-cert\"" Oct 29 22:38:30.359545 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.359511 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:38:30.365578 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.365536 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-75ea0c-openshift-default-888bbff8f-77bbd"] Oct 29 22:38:30.394337 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.394298 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndj9w\" (UniqueName: \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.394608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.394589 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.394710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.394633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.394874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.394852 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.496409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.496328 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndj9w\" (UniqueName: \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.496604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.496465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.496604 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.496502 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.496732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.496607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.497239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.497212 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.499106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.499075 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.499207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.499151 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.521645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.521616 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndj9w\" (UniqueName: \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\") pod \"maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.641053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.641021 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:30.960808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:30.960780 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:38:30.962789 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:38:30.962760 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533d11b2_d4bb_48eb_b462_81732dee78e9.slice/crio-f79420380ebcc080a79e1f34c854215328a58ad1b7f7b7890a823888cd11bfac WatchSource:0}: Error finding container f79420380ebcc080a79e1f34c854215328a58ad1b7f7b7890a823888cd11bfac: Status 404 returned error can't find the container with id f79420380ebcc080a79e1f34c854215328a58ad1b7f7b7890a823888cd11bfac Oct 29 22:38:31.300829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:31.300746 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b69987-5bf8-4949-82b0-7ca1f4e0eac3" path="/var/lib/kubelet/pods/12b69987-5bf8-4949-82b0-7ca1f4e0eac3/volumes" Oct 29 22:38:31.301573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:31.301529 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef2ee4d-cb06-4386-8909-c6f467f5d35d" path="/var/lib/kubelet/pods/aef2ee4d-cb06-4386-8909-c6f467f5d35d/volumes" Oct 29 22:38:31.309088 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:31.309059 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" event={"ID":"533d11b2-d4bb-48eb-b462-81732dee78e9","Type":"ContainerStarted","Data":"f79420380ebcc080a79e1f34c854215328a58ad1b7f7b7890a823888cd11bfac"} Oct 29 22:38:32.321608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:32.321574 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" event={"ID":"533d11b2-d4bb-48eb-b462-81732dee78e9","Type":"ContainerStarted","Data":"aa794dec46dea6b9b2a105daf5d76b93ccc08e621e0a3863303d71e1e0c2ab4b"} Oct 29 22:38:32.322031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:32.321639 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:32.370832 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:32.370776 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" podStartSLOduration=1.981250646 podStartE2EDuration="2.370756586s" totalImagesPullingTime="389.50594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:32.366913166 +0000 UTC m=+3371.215390003" watchObservedRunningTime="2026-10-29 22:38:32.370756586 +0000 UTC m=+3371.219233433" Oct 29 22:38:35.048876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:35.048790 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:38:36.106967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.106940 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:38:36.176856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.176810 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\" (UniqueName: \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\") pod \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " Oct 29 22:38:36.177222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.177195 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\") pod \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\" (UID: \"1e4373ca-fcb6-49f3-8b48-b735bab0c859\") " Oct 29 22:38:36.177751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.177722 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume" pod "1e4373ca-fcb6-49f3-8b48-b735bab0c859" (UID: "1e4373ca-fcb6-49f3-8b48-b735bab0c859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:36.178037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.178015 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4373ca-fcb6-49f3-8b48-b735bab0c859-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:36.179237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.179212 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz" pod "1e4373ca-fcb6-49f3-8b48-b735bab0c859" (UID: "1e4373ca-fcb6-49f3-8b48-b735bab0c859"). InnerVolumeSpecName "kube-api-access-chpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:36.279688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.279655 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpkz\" (UniqueName: \"kubernetes.io/projected/1e4373ca-fcb6-49f3-8b48-b735bab0c859-kube-api-access-chpkz\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:36.377422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.377318 4376 generic.go:356] "Generic (PLEG): container finished" podID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerID="75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0" exitCode=0 Oct 29 22:38:36.377422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.377361 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" Oct 29 22:38:36.377422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.377396 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" event={"ID":"1e4373ca-fcb6-49f3-8b48-b735bab0c859","Type":"ContainerDied","Data":"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0"} Oct 29 22:38:36.378200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.377452 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj" event={"ID":"1e4373ca-fcb6-49f3-8b48-b735bab0c859","Type":"ContainerDied","Data":"af0105e27e6f3924a648aa03d923725848d0fb5c36f1c42d0584d10dd571b73f"} Oct 29 22:38:36.378200 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.377475 4376 scope.go:122] "RemoveContainer" containerID="75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0" Oct 29 22:38:36.391242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.391219 4376 scope.go:122] "RemoveContainer" containerID="75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0" Oct 29 22:38:36.391530 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:38:36.391508 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0\": container with ID starting with 75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0 not found: ID does not exist" containerID="75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0" Oct 29 22:38:36.391575 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.391539 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0"} err="failed to get container status \"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0\": rpc error: code = NotFound desc = could not find container \"75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0\": container with ID starting with 75ec2ca779b34c64b4e2b823003efe75d0961da64cc962f30c9b8cd980b025c0 not found: ID does not exist" Oct 29 22:38:36.434842 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.434816 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:38:36.439503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:36.439474 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-50d4af-78956c7dcb-7mppj"] Oct 29 22:38:37.274930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:37.274867 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" path="/var/lib/kubelet/pods/1e4373ca-fcb6-49f3-8b48-b735bab0c859/volumes" Oct 29 22:38:38.336103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:38.336067 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:38:39.699803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:39.699770 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:38:39.700521 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:39.700376 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" podUID="6537394e-26ba-40f7-a956-1410208b0504" containerName="maas-api" containerID="cri-o://032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77" gracePeriod=30 Oct 29 22:38:40.068072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.065734 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" podUID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" containerName="payload-pre-processing" containerID="cri-o://13a036d757805ad8d427b0366ad7fd71fb1df90cbf43f3228782006a5c07141f" gracePeriod=25 Oct 29 22:38:40.221849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.221009 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:38:40.313685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.313584 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\" (UniqueName: \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\") pod \"6537394e-26ba-40f7-a956-1410208b0504\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " Oct 29 22:38:40.313869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.313681 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\") pod \"6537394e-26ba-40f7-a956-1410208b0504\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " Oct 29 22:38:40.313869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.313859 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\") pod \"6537394e-26ba-40f7-a956-1410208b0504\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " Oct 29 22:38:40.314956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.314086 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") pod \"6537394e-26ba-40f7-a956-1410208b0504\" (UID: \"6537394e-26ba-40f7-a956-1410208b0504\") " Oct 29 22:38:40.314956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.314406 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca" pod "6537394e-26ba-40f7-a956-1410208b0504" (UID: "6537394e-26ba-40f7-a956-1410208b0504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.315237 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.315207 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6537394e-26ba-40f7-a956-1410208b0504-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.316471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.316444 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs" pod "6537394e-26ba-40f7-a956-1410208b0504" (UID: "6537394e-26ba-40f7-a956-1410208b0504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:40.318327 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.318301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls" pod "6537394e-26ba-40f7-a956-1410208b0504" (UID: "6537394e-26ba-40f7-a956-1410208b0504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:40.321959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.321923 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t" pod "6537394e-26ba-40f7-a956-1410208b0504" (UID: "6537394e-26ba-40f7-a956-1410208b0504"). InnerVolumeSpecName "kube-api-access-sfw8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.417710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.417673 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.417997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.417977 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfw8t\" (UniqueName: \"kubernetes.io/projected/6537394e-26ba-40f7-a956-1410208b0504-kube-api-access-sfw8t\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.418117 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.418100 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6537394e-26ba-40f7-a956-1410208b0504-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.469888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.469851 4376 generic.go:356] "Generic (PLEG): container finished" podID="6537394e-26ba-40f7-a956-1410208b0504" containerID="032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77" exitCode=0 Oct 29 22:38:40.469888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.469917 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" Oct 29 22:38:40.470124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.469957 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" event={"ID":"6537394e-26ba-40f7-a956-1410208b0504","Type":"ContainerDied","Data":"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77"} Oct 29 22:38:40.470124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.470015 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl" event={"ID":"6537394e-26ba-40f7-a956-1410208b0504","Type":"ContainerDied","Data":"d70a64f9c8a4babc228e164480c40f10408f7958a6f19658b83f8afd41e76040"} Oct 29 22:38:40.470124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.470039 4376 scope.go:122] "RemoveContainer" containerID="032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77" Oct 29 22:38:40.471887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.471861 4376 generic.go:356] "Generic (PLEG): container finished" podID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" containerID="13a036d757805ad8d427b0366ad7fd71fb1df90cbf43f3228782006a5c07141f" exitCode=0 Oct 29 22:38:40.472011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.471952 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" event={"ID":"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd","Type":"ContainerDied","Data":"13a036d757805ad8d427b0366ad7fd71fb1df90cbf43f3228782006a5c07141f"} Oct 29 22:38:40.477867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.477829 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:38:40.482855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.482837 4376 scope.go:122] "RemoveContainer" containerID="032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77" Oct 29 22:38:40.483247 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:38:40.483222 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77\": container with ID starting with 032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77 not found: ID does not exist" containerID="032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77" Oct 29 22:38:40.483310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.483259 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77"} err="failed to get container status \"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77\": rpc error: code = NotFound desc = could not find container \"032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77\": container with ID starting with 032c9ec3f17e402af12550c59f5cbd8a241cabe1b86745da0fecc3d366aabc77 not found: ID does not exist" Oct 29 22:38:40.519285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.519252 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\" (UniqueName: \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\") pod \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " Oct 29 22:38:40.519514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.519376 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\") pod \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\" (UID: \"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd\") " Oct 29 22:38:40.520198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.520169 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume" pod "ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" (UID: "ea3870f7-ff9c-4408-9deb-9b2a09ec24fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.520388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.520371 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.521808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.521782 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr" pod "ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" (UID: "ea3870f7-ff9c-4408-9deb-9b2a09ec24fd"). InnerVolumeSpecName "kube-api-access-rkgwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.529170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.529140 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:38:40.534213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.534184 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-adfb9d-fdfd7f58f-dkjcl"] Oct 29 22:38:40.622619 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:40.622522 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgwr\" (UniqueName: \"kubernetes.io/projected/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd-kube-api-access-rkgwr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.107474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.107399 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb"] Oct 29 22:38:41.136423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.136351 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-adfb9d-lj9jb"] Oct 29 22:38:41.275210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.275177 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7645db-42a0-499c-8825-d190099f3f98" path="/var/lib/kubelet/pods/0d7645db-42a0-499c-8825-d190099f3f98/volumes" Oct 29 22:38:41.275741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.275725 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6537394e-26ba-40f7-a956-1410208b0504" path="/var/lib/kubelet/pods/6537394e-26ba-40f7-a956-1410208b0504/volumes" Oct 29 22:38:41.482817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.482783 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" Oct 29 22:38:41.483064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.482819 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652" event={"ID":"ea3870f7-ff9c-4408-9deb-9b2a09ec24fd","Type":"ContainerDied","Data":"2b9ed1c7b991b50f524e13828712dde5e91227e052249b4bdd1f96757225fef4"} Oct 29 22:38:41.483064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.482883 4376 scope.go:122] "RemoveContainer" containerID="13a036d757805ad8d427b0366ad7fd71fb1df90cbf43f3228782006a5c07141f" Oct 29 22:38:41.541233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.541205 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:38:41.553579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:41.553555 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26abe8-8597879f47-rb652"] Oct 29 22:38:43.275076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:38:43.275045 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" path="/var/lib/kubelet/pods/ea3870f7-ff9c-4408-9deb-9b2a09ec24fd/volumes" Oct 29 22:38:46.436737 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:38:46.436697 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:39:16.435221 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:16.435179 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:39:25.360930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:25.357063 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 22:39:25.360930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:25.357379 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" containerID="cri-o://4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d" gracePeriod=30 Oct 29 22:39:25.496197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:25.496160 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:39:25.496375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:25.496228 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:39:30.868878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.868844 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 22:39:30.980773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.980729 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.980807 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.980954 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.980985 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:30.981292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981122 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:30.981404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:30.981404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981292 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981449 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981500 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981538 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.981632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.981581 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\") pod \"e798caae-4b39-4079-8849-1a4bbae067ca\" (UID: \"e798caae-4b39-4079-8849-1a4bbae067ca\") " Oct 29 22:39:30.982517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982399 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:30.982717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982694 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:30.982797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:30.982797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982726 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:30.982797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982742 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:30.982797 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.982757 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:30.984085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.984012 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:30.984085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.984035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:30.984768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.984664 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:30.985186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:30.985162 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh" pod "e798caae-4b39-4079-8849-1a4bbae067ca" (UID: "e798caae-4b39-4079-8849-1a4bbae067ca"). InnerVolumeSpecName "kube-api-access-sjksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:31.086930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.086863 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.086930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.086935 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798caae-4b39-4079-8849-1a4bbae067ca-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.087344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.086952 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjksh\" (UniqueName: \"kubernetes.io/projected/e798caae-4b39-4079-8849-1a4bbae067ca-kube-api-access-sjksh\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.087344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.086966 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798caae-4b39-4079-8849-1a4bbae067ca-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.087344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.086983 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798caae-4b39-4079-8849-1a4bbae067ca-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.122404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.122350 4376 generic.go:356] "Generic (PLEG): container finished" podID="e798caae-4b39-4079-8849-1a4bbae067ca" containerID="4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d" exitCode=0 Oct 29 22:39:31.122636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.122413 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" Oct 29 22:39:31.122636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.122449 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" event={"ID":"e798caae-4b39-4079-8849-1a4bbae067ca","Type":"ContainerDied","Data":"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d"} Oct 29 22:39:31.122636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.122529 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2" event={"ID":"e798caae-4b39-4079-8849-1a4bbae067ca","Type":"ContainerDied","Data":"f6cf3a3d1b39f177d2763bee0284a3f338cfe2bdb69913e4f7f7dabf67ad7a1d"} Oct 29 22:39:31.122636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.122560 4376 scope.go:122] "RemoveContainer" containerID="4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d" Oct 29 22:39:31.140528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.140505 4376 scope.go:122] "RemoveContainer" containerID="4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d" Oct 29 22:39:31.140878 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:31.140850 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d\": container with ID starting with 4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d not found: ID does not exist" containerID="4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d" Oct 29 22:39:31.140988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.140885 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d"} err="failed to get container status \"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d\": rpc error: code = NotFound desc = could not find container \"4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d\": container with ID starting with 4e56f6d12004474c19784519c856497ab205bc6ba01fb7717b83d16e2c290d4d not found: ID does not exist" Oct 29 22:39:31.184297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.184252 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 22:39:31.195382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.195347 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-vftb2"] Oct 29 22:39:31.286479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:31.286377 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" path="/var/lib/kubelet/pods/e798caae-4b39-4079-8849-1a4bbae067ca/volumes" Oct 29 22:39:34.380463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:34.380412 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:39:34.381232 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:34.380727 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" podUID="db0c730c-60b9-4401-b27b-3f8459333391" containerName="maas-api" containerID="cri-o://f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0" gracePeriod=30 Oct 29 22:39:34.386589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:34.386352 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" podUID="db0c730c-60b9-4401-b27b-3f8459333391" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.202:8443/health\": dial tcp 10.134.0.202:8443: connect: connection refused" Oct 29 22:39:34.970372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:34.970318 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:39:35.136207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.136164 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") pod \"db0c730c-60b9-4401-b27b-3f8459333391\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " Oct 29 22:39:35.136599 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.136568 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\") pod \"db0c730c-60b9-4401-b27b-3f8459333391\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " Oct 29 22:39:35.137606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.137208 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\") pod \"db0c730c-60b9-4401-b27b-3f8459333391\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " Oct 29 22:39:35.138071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.138043 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca" pod "db0c730c-60b9-4401-b27b-3f8459333391" (UID: "db0c730c-60b9-4401-b27b-3f8459333391"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:35.138630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.138428 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\" (UniqueName: \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\") pod \"db0c730c-60b9-4401-b27b-3f8459333391\" (UID: \"db0c730c-60b9-4401-b27b-3f8459333391\") " Oct 29 22:39:35.139252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.139231 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0c730c-60b9-4401-b27b-3f8459333391-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:35.139868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.139843 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls" pod "db0c730c-60b9-4401-b27b-3f8459333391" (UID: "db0c730c-60b9-4401-b27b-3f8459333391"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:35.140649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.140614 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs" pod "db0c730c-60b9-4401-b27b-3f8459333391" (UID: "db0c730c-60b9-4401-b27b-3f8459333391"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:35.141071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.141048 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv" pod "db0c730c-60b9-4401-b27b-3f8459333391" (UID: "db0c730c-60b9-4401-b27b-3f8459333391"). InnerVolumeSpecName "kube-api-access-7przv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:35.164520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.164096 4376 generic.go:356] "Generic (PLEG): container finished" podID="db0c730c-60b9-4401-b27b-3f8459333391" containerID="f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0" exitCode=0 Oct 29 22:39:35.164520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.164395 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" event={"ID":"db0c730c-60b9-4401-b27b-3f8459333391","Type":"ContainerDied","Data":"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0"} Oct 29 22:39:35.164520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.164454 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" event={"ID":"db0c730c-60b9-4401-b27b-3f8459333391","Type":"ContainerDied","Data":"8aa680410295a402d59615ada4b150e4069373feec7784b2e892f0276a2386b5"} Oct 29 22:39:35.168984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.164486 4376 scope.go:122] "RemoveContainer" containerID="f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0" Oct 29 22:39:35.168984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.165241 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn" Oct 29 22:39:35.184979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.184949 4376 scope.go:122] "RemoveContainer" containerID="f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0" Oct 29 22:39:35.185348 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:35.185316 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0\": container with ID starting with f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0 not found: ID does not exist" containerID="f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0" Oct 29 22:39:35.185429 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.185363 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0"} err="failed to get container status \"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0\": rpc error: code = NotFound desc = could not find container \"f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0\": container with ID starting with f5b311d8a4c39a53f101d2d20e4db645047e7d822d7c0fb7a12ceb2a2d1d5af0 not found: ID does not exist" Oct 29 22:39:35.228940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.228886 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:39:35.234418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.234376 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-797e0c-cf498c9fb-qpgmn"] Oct 29 22:39:35.241061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.241018 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7przv\" (UniqueName: \"kubernetes.io/projected/db0c730c-60b9-4401-b27b-3f8459333391-kube-api-access-7przv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:35.241061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.241054 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:35.241061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.241066 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0c730c-60b9-4401-b27b-3f8459333391-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:35.277466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:35.277425 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0c730c-60b9-4401-b27b-3f8459333391" path="/var/lib/kubelet/pods/db0c730c-60b9-4401-b27b-3f8459333391/volumes" Oct 29 22:39:40.751917 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:40.751858 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:39:40.752454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:40.752213 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" podUID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" containerName="maas-api" containerID="cri-o://a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2" gracePeriod=30 Oct 29 22:39:41.295934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.249334 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:39:41.326411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.324851 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\" (UniqueName: \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\") pod \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " Oct 29 22:39:41.326411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.324976 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\") pod \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " Oct 29 22:39:41.326411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.325153 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\") pod \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " Oct 29 22:39:41.326411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.325207 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\") pod \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\" (UID: \"0e2b9788-57fe-4ce5-8386-80ea9d248a61\") " Oct 29 22:39:41.327341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.326982 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca" pod "0e2b9788-57fe-4ce5-8386-80ea9d248a61" (UID: "0e2b9788-57fe-4ce5-8386-80ea9d248a61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:41.356929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.354100 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs" pod "0e2b9788-57fe-4ce5-8386-80ea9d248a61" (UID: "0e2b9788-57fe-4ce5-8386-80ea9d248a61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:41.426931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.421132 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls" pod "0e2b9788-57fe-4ce5-8386-80ea9d248a61" (UID: "0e2b9788-57fe-4ce5-8386-80ea9d248a61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:41.426931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.421300 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p" pod "0e2b9788-57fe-4ce5-8386-80ea9d248a61" (UID: "0e2b9788-57fe-4ce5-8386-80ea9d248a61"). InnerVolumeSpecName "kube-api-access-hm88p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:41.457001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.456963 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:41.464880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.464848 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2b9788-57fe-4ce5-8386-80ea9d248a61-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:41.465161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.465142 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm88p\" (UniqueName: \"kubernetes.io/projected/0e2b9788-57fe-4ce5-8386-80ea9d248a61-kube-api-access-hm88p\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:41.465284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.465271 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2b9788-57fe-4ce5-8386-80ea9d248a61-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:41.487020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.470079 4376 generic.go:356] "Generic (PLEG): container finished" podID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" containerID="a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2" exitCode=0 Oct 29 22:39:41.487020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.470110 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" event={"ID":"0e2b9788-57fe-4ce5-8386-80ea9d248a61","Type":"ContainerDied","Data":"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2"} Oct 29 22:39:41.487020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.486325 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" event={"ID":"0e2b9788-57fe-4ce5-8386-80ea9d248a61","Type":"ContainerDied","Data":"660c6e8d1e55bf85aa229251febeb1597499eaadc0f2016292912bddffee341f"} Oct 29 22:39:41.487020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.486369 4376 scope.go:122] "RemoveContainer" containerID="a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2" Oct 29 22:39:41.487020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.470180 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc" Oct 29 22:39:41.534075 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.533928 4376 scope.go:122] "RemoveContainer" containerID="a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2" Oct 29 22:39:41.535825 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:41.534810 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2\": container with ID starting with a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2 not found: ID does not exist" containerID="a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2" Oct 29 22:39:41.535825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.534853 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2"} err="failed to get container status \"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2\": rpc error: code = NotFound desc = could not find container \"a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2\": container with ID starting with a76669fac8b2278302fa5bf47d4653afe81e819f0ff9b4a457b995293b88b2d2 not found: ID does not exist" Oct 29 22:39:41.557085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.557041 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:39:41.562579 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:41.562541 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-05687d-7fd967d6cd-dc7dc"] Oct 29 22:39:43.274656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:43.274622 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" path="/var/lib/kubelet/pods/0e2b9788-57fe-4ce5-8386-80ea9d248a61/volumes" Oct 29 22:39:45.711152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:45.711107 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:39:45.711705 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:45.711409 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" podUID="7d307021-fd9c-4162-a225-b5e628de2808" containerName="maas-api" containerID="cri-o://c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c" gracePeriod=30 Oct 29 22:39:46.245202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.245093 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:39:46.366109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.366063 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\") pod \"7d307021-fd9c-4162-a225-b5e628de2808\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " Oct 29 22:39:46.366466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.366296 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\") pod \"7d307021-fd9c-4162-a225-b5e628de2808\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " Oct 29 22:39:46.366550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.366528 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\" (UniqueName: \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\") pod \"7d307021-fd9c-4162-a225-b5e628de2808\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " Oct 29 22:39:46.368524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.366608 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\") pod \"7d307021-fd9c-4162-a225-b5e628de2808\" (UID: \"7d307021-fd9c-4162-a225-b5e628de2808\") " Oct 29 22:39:46.368524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.367856 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca" pod "7d307021-fd9c-4162-a225-b5e628de2808" (UID: "7d307021-fd9c-4162-a225-b5e628de2808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:46.373517 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.373435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k" pod "7d307021-fd9c-4162-a225-b5e628de2808" (UID: "7d307021-fd9c-4162-a225-b5e628de2808"). InnerVolumeSpecName "kube-api-access-2zj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:46.373861 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.373805 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs" pod "7d307021-fd9c-4162-a225-b5e628de2808" (UID: "7d307021-fd9c-4162-a225-b5e628de2808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:46.377481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.377393 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls" pod "7d307021-fd9c-4162-a225-b5e628de2808" (UID: "7d307021-fd9c-4162-a225-b5e628de2808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:46.456584 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:46.456544 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:39:46.469004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.468964 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj6k\" (UniqueName: \"kubernetes.io/projected/7d307021-fd9c-4162-a225-b5e628de2808-kube-api-access-2zj6k\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:46.469004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.469002 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d307021-fd9c-4162-a225-b5e628de2808-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:46.469004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.469013 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:46.469258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.469023 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d307021-fd9c-4162-a225-b5e628de2808-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:46.559542 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.559501 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d307021-fd9c-4162-a225-b5e628de2808" containerID="c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c" exitCode=0 Oct 29 22:39:46.559760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.559554 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" Oct 29 22:39:46.559760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.559588 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" event={"ID":"7d307021-fd9c-4162-a225-b5e628de2808","Type":"ContainerDied","Data":"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c"} Oct 29 22:39:46.559760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.559647 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b" event={"ID":"7d307021-fd9c-4162-a225-b5e628de2808","Type":"ContainerDied","Data":"84d276edb7d759ba2626cfa2c27ef2b082173a5fc554e71aea6964a3cae0b641"} Oct 29 22:39:46.559760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.559671 4376 scope.go:122] "RemoveContainer" containerID="c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c" Oct 29 22:39:46.571764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.571738 4376 scope.go:122] "RemoveContainer" containerID="c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c" Oct 29 22:39:46.572145 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:46.572120 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c\": container with ID starting with c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c not found: ID does not exist" containerID="c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c" Oct 29 22:39:46.572236 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.572155 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c"} err="failed to get container status \"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c\": rpc error: code = NotFound desc = could not find container \"c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c\": container with ID starting with c32d9b7b8a4f2985cbb45fc8c20c09e2bfd1eab32bc8955f7ba3ed2f73c8e29c not found: ID does not exist" Oct 29 22:39:46.617314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.617271 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:39:46.629502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:46.629460 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e7b56d-74c97dbc58-h5t9b"] Oct 29 22:39:47.275543 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:47.275506 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d307021-fd9c-4162-a225-b5e628de2808" path="/var/lib/kubelet/pods/7d307021-fd9c-4162-a225-b5e628de2808/volumes" Oct 29 22:39:49.937163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:49.937117 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:39:49.937735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:49.937443 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" containerID="cri-o://a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952" gracePeriod=30 Oct 29 22:39:51.364067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:51.364023 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:39:51.415333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:51.415280 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:39:55.430060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.430024 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:39:55.499472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499437 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.499683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499498 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.499683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499570 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.499683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.499880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499783 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.499880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499795 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.499880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499837 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.500082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.499935 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.500207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.500178 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.500286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.500249 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.500443 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.500415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.500589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.500472 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\") pod \"1093f24b-6824-47b6-a2fb-cedb867475b3\" (UID: \"1093f24b-6824-47b6-a2fb-cedb867475b3\") " Oct 29 22:39:55.500853 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.500828 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.501467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.501440 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.501580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.501473 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1093f24b-6824-47b6-a2fb-cedb867475b3-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.501580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.501489 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.501580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.501504 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.501823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.501801 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.502377 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.502354 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.502481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.502450 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "kube-api-access-8sqkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.502546 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.502521 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.502820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.502787 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo" pod "1093f24b-6824-47b6-a2fb-cedb867475b3" (UID: "1093f24b-6824-47b6-a2fb-cedb867475b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.603741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.603635 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.603741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.603675 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.603741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.603691 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqkl\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-kube-api-access-8sqkl\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.603741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.603704 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.603741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.603715 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1093f24b-6824-47b6-a2fb-cedb867475b3-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.719053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.719011 4376 generic.go:356] "Generic (PLEG): container finished" podID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerID="a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952" exitCode=0 Oct 29 22:39:55.719250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.719060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" event={"ID":"1093f24b-6824-47b6-a2fb-cedb867475b3","Type":"ContainerDied","Data":"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952"} Oct 29 22:39:55.719250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.719089 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" Oct 29 22:39:55.719250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.719124 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6" event={"ID":"1093f24b-6824-47b6-a2fb-cedb867475b3","Type":"ContainerDied","Data":"9bc42efd15147a857fb8c22729be934cdf1e00d2e16f2f345b8a1f1718bbca14"} Oct 29 22:39:55.719250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.719149 4376 scope.go:122] "RemoveContainer" containerID="a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952" Oct 29 22:39:55.732624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.732600 4376 scope.go:122] "RemoveContainer" containerID="a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952" Oct 29 22:39:55.732955 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:39:55.732934 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952\": container with ID starting with a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952 not found: ID does not exist" containerID="a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952" Oct 29 22:39:55.733039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.732969 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952"} err="failed to get container status \"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952\": rpc error: code = NotFound desc = could not find container \"a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952\": container with ID starting with a11a7709a7f9d06b87f5d9464cedea499204326aaab10ca569eceacce1b0a952 not found: ID does not exist" Oct 29 22:39:55.779992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.779950 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:39:55.790157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:55.790116 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-7fzr6"] Oct 29 22:39:56.365315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.365273 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" podUID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" containerName="payload-processing" containerID="cri-o://79c62799ee7997b36a12375e74918db3b3e4b3c845708e5b30392ec99d58200a" gracePeriod=25 Oct 29 22:39:56.416213 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.416165 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" podUID="7b17f744-3e78-4206-9817-ff7482e3ab49" containerName="payload-pre-processing" containerID="cri-o://444116aa9405a7628a2eac9a9ed681f495a942c7a0f79dceb8ea3eb15a21e1a5" gracePeriod=25 Oct 29 22:39:56.505635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.505593 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:39:56.562526 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.562463 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:39:56.734510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.734466 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b17f744-3e78-4206-9817-ff7482e3ab49" containerID="444116aa9405a7628a2eac9a9ed681f495a942c7a0f79dceb8ea3eb15a21e1a5" exitCode=0 Oct 29 22:39:56.734706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.734627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" event={"ID":"7b17f744-3e78-4206-9817-ff7482e3ab49","Type":"ContainerDied","Data":"444116aa9405a7628a2eac9a9ed681f495a942c7a0f79dceb8ea3eb15a21e1a5"} Oct 29 22:39:56.737711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.737673 4376 generic.go:356] "Generic (PLEG): container finished" podID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" containerID="79c62799ee7997b36a12375e74918db3b3e4b3c845708e5b30392ec99d58200a" exitCode=0 Oct 29 22:39:56.737974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.737896 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" event={"ID":"dbea50e9-6d66-4ae1-ba71-dd29b98795cc","Type":"ContainerDied","Data":"79c62799ee7997b36a12375e74918db3b3e4b3c845708e5b30392ec99d58200a"} Oct 29 22:39:56.885034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.884997 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:39:56.892686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.892420 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:39:56.965976 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.965932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\") pod \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " Oct 29 22:39:56.966207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.966059 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\") pod \"7b17f744-3e78-4206-9817-ff7482e3ab49\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " Oct 29 22:39:56.966207 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.966154 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\" (UniqueName: \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\") pod \"7b17f744-3e78-4206-9817-ff7482e3ab49\" (UID: \"7b17f744-3e78-4206-9817-ff7482e3ab49\") " Oct 29 22:39:56.967827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.966332 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\" (UniqueName: \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\") pod \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\" (UID: \"dbea50e9-6d66-4ae1-ba71-dd29b98795cc\") " Oct 29 22:39:56.967827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.966328 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume" pod "dbea50e9-6d66-4ae1-ba71-dd29b98795cc" (UID: "dbea50e9-6d66-4ae1-ba71-dd29b98795cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:56.967827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.966740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume" pod "7b17f744-3e78-4206-9817-ff7482e3ab49" (UID: "7b17f744-3e78-4206-9817-ff7482e3ab49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:56.971657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.970674 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk" pod "dbea50e9-6d66-4ae1-ba71-dd29b98795cc" (UID: "dbea50e9-6d66-4ae1-ba71-dd29b98795cc"). InnerVolumeSpecName "kube-api-access-cqjdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:56.971657 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:56.970796 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4" pod "7b17f744-3e78-4206-9817-ff7482e3ab49" (UID: "7b17f744-3e78-4206-9817-ff7482e3ab49"). InnerVolumeSpecName "kube-api-access-8h5f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:57.068771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.068724 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5f4\" (UniqueName: \"kubernetes.io/projected/7b17f744-3e78-4206-9817-ff7482e3ab49-kube-api-access-8h5f4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:57.068771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.068761 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjdk\" (UniqueName: \"kubernetes.io/projected/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-kube-api-access-cqjdk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:57.068771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.068771 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea50e9-6d66-4ae1-ba71-dd29b98795cc-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:57.069079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.068797 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b17f744-3e78-4206-9817-ff7482e3ab49-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:57.354984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.353273 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" path="/var/lib/kubelet/pods/1093f24b-6824-47b6-a2fb-cedb867475b3/volumes" Oct 29 22:39:57.749669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.749639 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" Oct 29 22:39:57.749669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.749654 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262" event={"ID":"dbea50e9-6d66-4ae1-ba71-dd29b98795cc","Type":"ContainerDied","Data":"146bca5869ccc4d0e5703459157d9adcc0a422b3d23c6b4716d022f362166454"} Oct 29 22:39:57.750297 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.749755 4376 scope.go:122] "RemoveContainer" containerID="79c62799ee7997b36a12375e74918db3b3e4b3c845708e5b30392ec99d58200a" Oct 29 22:39:57.751785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.751756 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" event={"ID":"7b17f744-3e78-4206-9817-ff7482e3ab49","Type":"ContainerDied","Data":"b6cfae7f23547e7aeca4a10d10e11d4c7cca5219e793fd2acd21a96ef2084f7b"} Oct 29 22:39:57.751785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.751782 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj" Oct 29 22:39:57.761732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.761704 4376 scope.go:122] "RemoveContainer" containerID="444116aa9405a7628a2eac9a9ed681f495a942c7a0f79dceb8ea3eb15a21e1a5" Oct 29 22:39:57.808934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.808867 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:39:57.820640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.820601 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7fe78e-5698c4559b-2l262"] Oct 29 22:39:57.857376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.857335 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:39:57.862822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:57.862784 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fe78e-5bb5d86979-mvwfj"] Oct 29 22:39:59.276809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:59.276772 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b17f744-3e78-4206-9817-ff7482e3ab49" path="/var/lib/kubelet/pods/7b17f744-3e78-4206-9817-ff7482e3ab49/volumes" Oct 29 22:39:59.277341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:39:59.277323 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" path="/var/lib/kubelet/pods/dbea50e9-6d66-4ae1-ba71-dd29b98795cc/volumes" Oct 29 22:40:00.953430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:00.953380 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:40:01.515886 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:01.515834 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" containerID="cri-o://c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469" gracePeriod=25 Oct 29 22:40:01.563508 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:01.563436 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" containerID="cri-o://445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b" gracePeriod=25 Oct 29 22:40:04.199613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:04.199523 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:04.199613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:04.199554 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:05.291403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.291354 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:40:05.291963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.291605 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" podUID="3b9909e5-b774-4720-858b-1c6271e8ac5c" containerName="maas-api" containerID="cri-o://4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a" gracePeriod=30 Oct 29 22:40:05.862747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.862709 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:40:05.887502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.887453 4376 generic.go:356] "Generic (PLEG): container finished" podID="3b9909e5-b774-4720-858b-1c6271e8ac5c" containerID="4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a" exitCode=0 Oct 29 22:40:05.888192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.888154 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" event={"ID":"3b9909e5-b774-4720-858b-1c6271e8ac5c","Type":"ContainerDied","Data":"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a"} Oct 29 22:40:05.888430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.888397 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" event={"ID":"3b9909e5-b774-4720-858b-1c6271e8ac5c","Type":"ContainerDied","Data":"31c721ef159ef42afa4517d2bc6f6bebec335f16f8e8f92ceb809970f2f48782"} Oct 29 22:40:05.888566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.888552 4376 scope.go:122] "RemoveContainer" containerID="4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a" Oct 29 22:40:05.888869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.888854 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n" Oct 29 22:40:05.911966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.911880 4376 scope.go:122] "RemoveContainer" containerID="4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a" Oct 29 22:40:05.912352 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:05.912304 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a\": container with ID starting with 4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a not found: ID does not exist" containerID="4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a" Oct 29 22:40:05.912433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.912363 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a"} err="failed to get container status \"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a\": rpc error: code = NotFound desc = could not find container \"4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a\": container with ID starting with 4c255b2d861a3f89ddc87ccab21cb12a57c51ea85c842bd6ce9af1be92db290a not found: ID does not exist" Oct 29 22:40:05.945896 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.945851 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\") pod \"3b9909e5-b774-4720-858b-1c6271e8ac5c\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " Oct 29 22:40:05.946130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.946101 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\") pod \"3b9909e5-b774-4720-858b-1c6271e8ac5c\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " Oct 29 22:40:05.946317 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.946299 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\") pod \"3b9909e5-b774-4720-858b-1c6271e8ac5c\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " Oct 29 22:40:05.946417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.946365 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\" (UniqueName: \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\") pod \"3b9909e5-b774-4720-858b-1c6271e8ac5c\" (UID: \"3b9909e5-b774-4720-858b-1c6271e8ac5c\") " Oct 29 22:40:05.946966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.946887 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca" pod "3b9909e5-b774-4720-858b-1c6271e8ac5c" (UID: "3b9909e5-b774-4720-858b-1c6271e8ac5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:05.947286 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.947264 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9909e5-b774-4720-858b-1c6271e8ac5c-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:05.948989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.948959 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs" pod "3b9909e5-b774-4720-858b-1c6271e8ac5c" (UID: "3b9909e5-b774-4720-858b-1c6271e8ac5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:05.949273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.949246 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls" pod "3b9909e5-b774-4720-858b-1c6271e8ac5c" (UID: "3b9909e5-b774-4720-858b-1c6271e8ac5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:05.949461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.949440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx" pod "3b9909e5-b774-4720-858b-1c6271e8ac5c" (UID: "3b9909e5-b774-4720-858b-1c6271e8ac5c"). InnerVolumeSpecName "kube-api-access-tj5cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:05.954612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:05.954567 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" containerID="cri-o://2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea" gracePeriod=25 Oct 29 22:40:06.049242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:06.049197 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5cx\" (UniqueName: \"kubernetes.io/projected/3b9909e5-b774-4720-858b-1c6271e8ac5c-kube-api-access-tj5cx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:06.049242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:06.049238 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:06.049242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:06.049248 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9909e5-b774-4720-858b-1c6271e8ac5c-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:06.244306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:06.244271 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:40:06.260692 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:06.260647 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-76c1b7-8cfdc48c9-pmq9n"] Oct 29 22:40:07.275116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:07.275079 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9909e5-b774-4720-858b-1c6271e8ac5c" path="/var/lib/kubelet/pods/3b9909e5-b774-4720-858b-1c6271e8ac5c/volumes" Oct 29 22:40:08.902998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.902954 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:40:08.903915 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903878 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3b9909e5-b774-4720-858b-1c6271e8ac5c" containerName="maas-api" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903919 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9909e5-b774-4720-858b-1c6271e8ac5c" containerName="maas-api" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903936 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6537394e-26ba-40f7-a956-1410208b0504" containerName="maas-api" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903942 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6537394e-26ba-40f7-a956-1410208b0504" containerName="maas-api" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903976 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903982 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903991 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" containerName="payload-processing" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.903999 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" containerName="payload-processing" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904009 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b17f744-3e78-4206-9817-ff7482e3ab49" containerName="payload-pre-processing" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904014 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b17f744-3e78-4206-9817-ff7482e3ab49" containerName="payload-pre-processing" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904028 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7d307021-fd9c-4162-a225-b5e628de2808" containerName="maas-api" Oct 29 22:40:08.904037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904034 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d307021-fd9c-4162-a225-b5e628de2808" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904050 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904055 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904066 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904071 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904083 4376 cpu_manager.go:414] "containerMap: removing container" podUID="db0c730c-60b9-4401-b27b-3f8459333391" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904087 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0c730c-60b9-4401-b27b-3f8459333391" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904097 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904101 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904110 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" containerName="payload-pre-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904115 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" containerName="payload-pre-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904258 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e798caae-4b39-4079-8849-1a4bbae067ca" containerName="istio-proxy" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904267 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbea50e9-6d66-4ae1-ba71-dd29b98795cc" containerName="payload-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904275 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b17f744-3e78-4206-9817-ff7482e3ab49" containerName="payload-pre-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904283 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d307021-fd9c-4162-a225-b5e628de2808" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904291 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2b9788-57fe-4ce5-8386-80ea9d248a61" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904297 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1093f24b-6824-47b6-a2fb-cedb867475b3" containerName="istio-proxy" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904303 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0c730c-60b9-4401-b27b-3f8459333391" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904309 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4373ca-fcb6-49f3-8b48-b735bab0c859" containerName="payload-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904315 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3870f7-ff9c-4408-9deb-9b2a09ec24fd" containerName="payload-pre-processing" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904322 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9909e5-b774-4720-858b-1c6271e8ac5c" containerName="maas-api" Oct 29 22:40:08.904592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.904327 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6537394e-26ba-40f7-a956-1410208b0504" containerName="maas-api" Oct 29 22:40:08.909399 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.909375 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:08.927602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.927562 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:40:08.933971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:08.933935 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-580af2-serving-cert\"" Oct 29 22:40:09.014205 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.014161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.014757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.014724 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.014960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.014917 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gdq\" (UniqueName: \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.015065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.014979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.116367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.116328 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.116593 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.116577 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.116694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.116670 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gdq\" (UniqueName: \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.116764 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.116715 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.117845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.117813 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.119624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.119600 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.119738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.119700 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.142159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.142123 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gdq\" (UniqueName: \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\") pod \"maas-api-e2e-inf-b-580af2-58889c679-7xfnm\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.236932 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.233275 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:09.595737 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.595702 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:40:09.596843 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:40:09.596811 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2fba5c_c2b7_44d1_87fd_12d623f8cc87.slice/crio-b1ddbd9bb3aaf64b31a91660aea32a9a8b95c0d72dbb77f4551de815e04fe3d8 WatchSource:0}: Error finding container b1ddbd9bb3aaf64b31a91660aea32a9a8b95c0d72dbb77f4551de815e04fe3d8: Status 404 returned error can't find the container with id b1ddbd9bb3aaf64b31a91660aea32a9a8b95c0d72dbb77f4551de815e04fe3d8 Oct 29 22:40:09.991243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:09.991201 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" event={"ID":"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87","Type":"ContainerStarted","Data":"b1ddbd9bb3aaf64b31a91660aea32a9a8b95c0d72dbb77f4551de815e04fe3d8"} Oct 29 22:40:11.000813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:11.000767 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" event={"ID":"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87","Type":"ContainerStarted","Data":"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a"} Oct 29 22:40:11.001306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:11.001037 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:11.046466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:11.046404 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" podStartSLOduration=2.593746608 podStartE2EDuration="3.046375517s" totalImagesPullingTime="452.628909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:11.044628398 +0000 UTC m=+3469.893105247" watchObservedRunningTime="2026-10-29 22:40:11.046375517 +0000 UTC m=+3469.894852364" Oct 29 22:40:14.201929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:14.200897 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:14.201929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:14.201261 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:15.575875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:15.575794 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:15.667462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:15.667428 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:40:15.667664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:15.667643 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" containerID="cri-o://8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047" gracePeriod=30 Oct 29 22:40:16.046476 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.046433 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:40:16.062138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.062104 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:40:16.062345 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.062236 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.081401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.081374 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-563f39\"/\"default-dockercfg-sx5cd\"" Oct 29 22:40:16.081540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.081376 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-563f39\"/\"openshift-service-ca.crt\"" Oct 29 22:40:16.081626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.081537 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-563f39\"/\"test-model-563f39-kserve-self-signed-certs\"" Oct 29 22:40:16.081626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.081388 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-563f39\"/\"kube-root-ca.crt\"" Oct 29 22:40:16.170043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170007 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.170223 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170109 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.170267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.170306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170270 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.170346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170332 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bznk\" (UniqueName: \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.170484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.170463 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.283602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.283553 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.283802 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.283630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.283868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.283811 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.283868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.283849 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.287926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.287005 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.287926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.287593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.289791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.289729 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.293914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.290934 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bznk\" (UniqueName: \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.293914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.291129 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.293914 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.291637 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.294749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.294724 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.342157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.342076 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bznk\" (UniqueName: \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\") pod \"test-model-563f39-kserve-dfbc666cf-9kt8f\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.382952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.381725 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:16.450242 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:16.449550 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:40:16.743645 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:16.743610 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:40:16.745006 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:40:16.744970 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc354ed1c_da1b_459f_b7b1_4089adb4d83d.slice/crio-a18d2c15329d6c7c707cd67021f9d54421982620348fd19428f29cf15f38fa01 WatchSource:0}: Error finding container a18d2c15329d6c7c707cd67021f9d54421982620348fd19428f29cf15f38fa01: Status 404 returned error can't find the container with id a18d2c15329d6c7c707cd67021f9d54421982620348fd19428f29cf15f38fa01 Oct 29 22:40:17.015084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.015001 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:40:17.068010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.067962 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" event={"ID":"c354ed1c-da1b-459f-b7b1-4089adb4d83d","Type":"ContainerStarted","Data":"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f"} Oct 29 22:40:17.068192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.068017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" event={"ID":"c354ed1c-da1b-459f-b7b1-4089adb4d83d","Type":"ContainerStarted","Data":"a18d2c15329d6c7c707cd67021f9d54421982620348fd19428f29cf15f38fa01"} Oct 29 22:40:17.113715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.113651 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" podStartSLOduration=1.113635396 podStartE2EDuration="1.113635396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:17.111551396 +0000 UTC m=+3475.960028276" watchObservedRunningTime="2026-10-29 22:40:17.113635396 +0000 UTC m=+3475.962112246" Oct 29 22:40:17.382319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.382227 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:17.884529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.884479 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:40:17.885443 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:17.884770 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" containerID="cri-o://6da5a7c83029cb0cc3b7e8c4f65e4a1823c04643e75fc4e200cdd5d6a1fd5aa6" gracePeriod=30 Oct 29 22:40:20.977625 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.977594 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:40:20.990647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.990608 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.990831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.990703 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.990876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.990862 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.990959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.990942 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991077 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991149 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991166 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:20.991303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991244 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:20.991303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991269 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991369 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:20.991431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991351 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991443 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\") pod \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\" (UID: \"dc5f92d3-8c87-4b8d-b443-d2d788fb2957\") " Oct 29 22:40:20.991789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991762 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:20.991919 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.991733 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:20.992328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.992298 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.992328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.992327 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.992515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.992336 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.992515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.992345 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.992515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.992354 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.993168 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.993142 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:20.993349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.993318 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:20.993786 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.993762 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "kube-api-access-2sg5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:20.993850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:20.993799 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy" pod "dc5f92d3-8c87-4b8d-b443-d2d788fb2957" (UID: "dc5f92d3-8c87-4b8d-b443-d2d788fb2957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:21.093869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.093827 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sg5b\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-kube-api-access-2sg5b\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:21.093869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.093867 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:21.093869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.093879 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:21.094143 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.093890 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5f92d3-8c87-4b8d-b443-d2d788fb2957-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:21.134304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.134266 4376 generic.go:356] "Generic (PLEG): container finished" podID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerID="8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047" exitCode=0 Oct 29 22:40:21.134507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.134311 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" event={"ID":"dc5f92d3-8c87-4b8d-b443-d2d788fb2957","Type":"ContainerDied","Data":"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047"} Oct 29 22:40:21.134507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.134316 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" Oct 29 22:40:21.134507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.134397 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb" event={"ID":"dc5f92d3-8c87-4b8d-b443-d2d788fb2957","Type":"ContainerDied","Data":"89c02eb57a4423f93a47a285de3710561365904709d42ef337e93850942b8c41"} Oct 29 22:40:21.134507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.134423 4376 scope.go:122] "RemoveContainer" containerID="8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047" Oct 29 22:40:21.152289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.152232 4376 scope.go:122] "RemoveContainer" containerID="8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047" Oct 29 22:40:21.154446 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:21.154356 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047\": container with ID starting with 8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047 not found: ID does not exist" containerID="8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047" Oct 29 22:40:21.154446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.154421 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047"} err="failed to get container status \"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047\": rpc error: code = NotFound desc = could not find container \"8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047\": container with ID starting with 8f4de2ce6b0d9c09c26c0caf2bf8cf6a60677473bce60e57947b6e9131d5f047 not found: ID does not exist" Oct 29 22:40:21.196148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.196062 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:40:21.203870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.203833 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-wmtjb"] Oct 29 22:40:21.276185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.276151 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" path="/var/lib/kubelet/pods/dc5f92d3-8c87-4b8d-b443-d2d788fb2957/volumes" Oct 29 22:40:21.872827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.872787 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:40:21.907825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.907788 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\" (UniqueName: \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\") pod \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " Oct 29 22:40:21.908276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.908246 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\") pod \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\" (UID: \"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad\") " Oct 29 22:40:21.908719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.908679 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume" pod "dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" (UID: "dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:21.909111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.909090 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:21.910246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.910226 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf" pod "dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" (UID: "dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad"). InnerVolumeSpecName "kube-api-access-4sxmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:21.935950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:21.935882 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:40:22.010000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.009944 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\") pod \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " Oct 29 22:40:22.011255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.010092 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\" (UniqueName: \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\") pod \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\" (UID: \"76fd685a-3c40-483e-b6fc-5087ec6ce75b\") " Oct 29 22:40:22.011255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.010769 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume" pod "76fd685a-3c40-483e-b6fc-5087ec6ce75b" (UID: "76fd685a-3c40-483e-b6fc-5087ec6ce75b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:22.011255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.011061 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fd685a-3c40-483e-b6fc-5087ec6ce75b-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.011255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.011098 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxmf\" (UniqueName: \"kubernetes.io/projected/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad-kube-api-access-4sxmf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.013308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.013278 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz" pod "76fd685a-3c40-483e-b6fc-5087ec6ce75b" (UID: "76fd685a-3c40-483e-b6fc-5087ec6ce75b"). InnerVolumeSpecName "kube-api-access-kflkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:22.112765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.112675 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflkz\" (UniqueName: \"kubernetes.io/projected/76fd685a-3c40-483e-b6fc-5087ec6ce75b-kube-api-access-kflkz\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.147718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.147678 4376 generic.go:356] "Generic (PLEG): container finished" podID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerID="445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b" exitCode=0 Oct 29 22:40:22.148039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.147739 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" Oct 29 22:40:22.148039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.147759 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" event={"ID":"76fd685a-3c40-483e-b6fc-5087ec6ce75b","Type":"ContainerDied","Data":"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b"} Oct 29 22:40:22.148039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.147821 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b" event={"ID":"76fd685a-3c40-483e-b6fc-5087ec6ce75b","Type":"ContainerDied","Data":"f9531ea29457565874e0f1aca58c73850dac738ba6d25bb723dd6b45cc547611"} Oct 29 22:40:22.148039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.147854 4376 scope.go:122] "RemoveContainer" containerID="445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b" Oct 29 22:40:22.149689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.149660 4376 generic.go:356] "Generic (PLEG): container finished" podID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerID="c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469" exitCode=0 Oct 29 22:40:22.149820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.149715 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" Oct 29 22:40:22.149820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.149741 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" event={"ID":"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad","Type":"ContainerDied","Data":"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469"} Oct 29 22:40:22.149820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.149798 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw" event={"ID":"dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad","Type":"ContainerDied","Data":"408a81cb2cfe9276bccc5dd6c1ed21064044b1cc78546b52751909e0f6417f33"} Oct 29 22:40:22.161897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.161872 4376 scope.go:122] "RemoveContainer" containerID="445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b" Oct 29 22:40:22.162456 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:22.162424 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b\": container with ID starting with 445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b not found: ID does not exist" containerID="445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b" Oct 29 22:40:22.162580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.162470 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b"} err="failed to get container status \"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b\": rpc error: code = NotFound desc = could not find container \"445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b\": container with ID starting with 445ce89c52437568f3e0a0572a787cc0d79dac255b4476779c9ffb1bd6b2cb9b not found: ID does not exist" Oct 29 22:40:22.162580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.162496 4376 scope.go:122] "RemoveContainer" containerID="c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469" Oct 29 22:40:22.178701 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.178674 4376 scope.go:122] "RemoveContainer" containerID="c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469" Oct 29 22:40:22.179047 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:22.179023 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469\": container with ID starting with c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469 not found: ID does not exist" containerID="c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469" Oct 29 22:40:22.179115 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.179057 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469"} err="failed to get container status \"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469\": rpc error: code = NotFound desc = could not find container \"c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469\": container with ID starting with c4d8b54d20326e5e0314861152d066ec733fd611de23c35009c900d2fa253469 not found: ID does not exist" Oct 29 22:40:22.206383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.206340 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:40:22.212304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.212272 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bef83b-7f66587dcf-czz2b"] Oct 29 22:40:22.246683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.246639 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:40:22.255319 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:22.255282 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bef83b-876b8c67b-qtbfw"] Oct 29 22:40:23.162464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.162426 4376 generic.go:356] "Generic (PLEG): container finished" podID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerID="6da5a7c83029cb0cc3b7e8c4f65e4a1823c04643e75fc4e200cdd5d6a1fd5aa6" exitCode=0 Oct 29 22:40:23.162931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.162500 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerDied","Data":"6da5a7c83029cb0cc3b7e8c4f65e4a1823c04643e75fc4e200cdd5d6a1fd5aa6"} Oct 29 22:40:23.162931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.162568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" event={"ID":"a27fa9f4-2bc6-406a-95c3-959ec1d2361d","Type":"ContainerDied","Data":"0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717"} Oct 29 22:40:23.162931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.162595 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f96550fe66cb77657445283207013a790a11fa3b49eb7a6f6f2a6c59ca61717" Oct 29 22:40:23.162931 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.162614 4376 scope.go:122] "RemoveContainer" containerID="2d4cde3911e106537ab83dfbd84748ade7c4e51f1898a33300683f24ca837bbc" Oct 29 22:40:23.184278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.184255 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:40:23.231211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231172 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231397 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231278 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231469 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231397 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231491 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231547 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231661 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231690 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.231868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231761 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:23.231868 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231788 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\") pod \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\" (UID: \"a27fa9f4-2bc6-406a-95c3-959ec1d2361d\") " Oct 29 22:40:23.232042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231844 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:23.232042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231893 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:23.232042 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.231979 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:23.232233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232210 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:23.232667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232645 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.232667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232668 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.232854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232678 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.232854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232687 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.232854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.232698 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.233935 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.233880 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:23.234080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.233947 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "kube-api-access-lhm4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:23.234080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.234009 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:23.234209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.234142 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy" pod "a27fa9f4-2bc6-406a-95c3-959ec1d2361d" (UID: "a27fa9f4-2bc6-406a-95c3-959ec1d2361d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:23.276238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.276148 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" path="/var/lib/kubelet/pods/76fd685a-3c40-483e-b6fc-5087ec6ce75b/volumes" Oct 29 22:40:23.276724 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.276708 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" path="/var/lib/kubelet/pods/dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad/volumes" Oct 29 22:40:23.334693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.334658 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.334693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.334688 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhm4t\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-kube-api-access-lhm4t\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.334693 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.334698 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.334984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:23.334707 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27fa9f4-2bc6-406a-95c3-959ec1d2361d-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:24.173201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:24.173166 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh" Oct 29 22:40:24.274769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:24.274729 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:40:24.281870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:24.281834 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-7fe78e-openshift-default-656774d58f-xdhxh"] Oct 29 22:40:25.275289 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:25.275253 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" path="/var/lib/kubelet/pods/a27fa9f4-2bc6-406a-95c3-959ec1d2361d/volumes" Oct 29 22:40:25.310071 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:25.310039 4376 scope.go:122] "RemoveContainer" containerID="6da5a7c83029cb0cc3b7e8c4f65e4a1823c04643e75fc4e200cdd5d6a1fd5aa6" Oct 29 22:40:25.576165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:25.576062 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:26.180550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.180520 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:40:26.190095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.190066 4376 generic.go:356] "Generic (PLEG): container finished" podID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerID="2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea" exitCode=0 Oct 29 22:40:26.190229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.190111 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" Oct 29 22:40:26.190229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.190151 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" event={"ID":"dbb4b42e-a45d-4576-abb2-629dd96ff906","Type":"ContainerDied","Data":"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea"} Oct 29 22:40:26.190229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.190212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc" event={"ID":"dbb4b42e-a45d-4576-abb2-629dd96ff906","Type":"ContainerDied","Data":"2e0387c53cea251e0250e5101a285dc7fea0021e8feb96ed36b2fb48713c0049"} Oct 29 22:40:26.190366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.190248 4376 scope.go:122] "RemoveContainer" containerID="2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea" Oct 29 22:40:26.201265 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.201233 4376 scope.go:122] "RemoveContainer" containerID="2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea" Oct 29 22:40:26.201582 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:26.201562 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea\": container with ID starting with 2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea not found: ID does not exist" containerID="2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea" Oct 29 22:40:26.201666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.201593 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea"} err="failed to get container status \"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea\": rpc error: code = NotFound desc = could not find container \"2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea\": container with ID starting with 2fd34971fc80cde18fbbc16696cedb1afa3d7966346120dadf71116be5872aea not found: ID does not exist" Oct 29 22:40:26.286781 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.286687 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\") pod \"dbb4b42e-a45d-4576-abb2-629dd96ff906\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " Oct 29 22:40:26.287362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.287135 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume" pod "dbb4b42e-a45d-4576-abb2-629dd96ff906" (UID: "dbb4b42e-a45d-4576-abb2-629dd96ff906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:26.287362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.287160 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\" (UniqueName: \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\") pod \"dbb4b42e-a45d-4576-abb2-629dd96ff906\" (UID: \"dbb4b42e-a45d-4576-abb2-629dd96ff906\") " Oct 29 22:40:26.287999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.287973 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb4b42e-a45d-4576-abb2-629dd96ff906-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:26.289362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.289334 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr" pod "dbb4b42e-a45d-4576-abb2-629dd96ff906" (UID: "dbb4b42e-a45d-4576-abb2-629dd96ff906"). InnerVolumeSpecName "kube-api-access-jb7lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:26.382890 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.382843 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:26.390263 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.390229 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb7lr\" (UniqueName: \"kubernetes.io/projected/dbb4b42e-a45d-4576-abb2-629dd96ff906-kube-api-access-jb7lr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:26.402394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.402362 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:26.616739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.616699 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:40:26.622185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:26.622154 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-76c1b7-7dfc4955f5-ltccc"] Oct 29 22:40:27.210262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:27.210233 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:40:27.276730 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:27.276691 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" path="/var/lib/kubelet/pods/dbb4b42e-a45d-4576-abb2-629dd96ff906/volumes" Oct 29 22:40:28.953766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:28.953730 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:40:28.988393 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:28.988360 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:40:33.954179 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:33.954138 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" containerID="cri-o://758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3" gracePeriod=25 Oct 29 22:40:33.989628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:33.989588 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" containerID="cri-o://7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250" gracePeriod=25 Oct 29 22:40:37.437589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:37.437546 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:37.438025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:37.437631 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:39.531421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:39.531383 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:40:39.531970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:39.531626 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" podUID="540a0d3b-7030-49ec-bb4e-ba95228a7228" containerName="maas-api" containerID="cri-o://3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8" gracePeriod=30 Oct 29 22:40:40.048922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.045724 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:40:40.146813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.146734 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\" (UniqueName: \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\") pod \"540a0d3b-7030-49ec-bb4e-ba95228a7228\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " Oct 29 22:40:40.146813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.146797 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\") pod \"540a0d3b-7030-49ec-bb4e-ba95228a7228\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " Oct 29 22:40:40.147043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.146859 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\") pod \"540a0d3b-7030-49ec-bb4e-ba95228a7228\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " Oct 29 22:40:40.147043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.146890 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\") pod \"540a0d3b-7030-49ec-bb4e-ba95228a7228\" (UID: \"540a0d3b-7030-49ec-bb4e-ba95228a7228\") " Oct 29 22:40:40.147250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.147216 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca" pod "540a0d3b-7030-49ec-bb4e-ba95228a7228" (UID: "540a0d3b-7030-49ec-bb4e-ba95228a7228"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:40.147522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.147499 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540a0d3b-7030-49ec-bb4e-ba95228a7228-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.149214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.149183 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls" pod "540a0d3b-7030-49ec-bb4e-ba95228a7228" (UID: "540a0d3b-7030-49ec-bb4e-ba95228a7228"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:40.149214 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.149205 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs" pod "540a0d3b-7030-49ec-bb4e-ba95228a7228" (UID: "540a0d3b-7030-49ec-bb4e-ba95228a7228"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:40.149348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.149287 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx" pod "540a0d3b-7030-49ec-bb4e-ba95228a7228" (UID: "540a0d3b-7030-49ec-bb4e-ba95228a7228"). InnerVolumeSpecName "kube-api-access-fwfdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:40.249020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.248986 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfdx\" (UniqueName: \"kubernetes.io/projected/540a0d3b-7030-49ec-bb4e-ba95228a7228-kube-api-access-fwfdx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.249020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.249016 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.249020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.249026 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540a0d3b-7030-49ec-bb4e-ba95228a7228-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.328233 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.328196 4376 generic.go:356] "Generic (PLEG): container finished" podID="540a0d3b-7030-49ec-bb4e-ba95228a7228" containerID="3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8" exitCode=0 Oct 29 22:40:40.328388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.328247 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" Oct 29 22:40:40.328388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.328275 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" event={"ID":"540a0d3b-7030-49ec-bb4e-ba95228a7228","Type":"ContainerDied","Data":"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8"} Oct 29 22:40:40.328388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.328333 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf" event={"ID":"540a0d3b-7030-49ec-bb4e-ba95228a7228","Type":"ContainerDied","Data":"0f1173b65e580ff9e56e3245291ce4e6d306e59e44734e9f682916afed7c6eaa"} Oct 29 22:40:40.328388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.328356 4376 scope.go:122] "RemoveContainer" containerID="3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8" Oct 29 22:40:40.339101 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.339084 4376 scope.go:122] "RemoveContainer" containerID="3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8" Oct 29 22:40:40.339354 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:40.339336 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8\": container with ID starting with 3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8 not found: ID does not exist" containerID="3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8" Oct 29 22:40:40.339398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.339362 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8"} err="failed to get container status \"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8\": rpc error: code = NotFound desc = could not find container \"3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8\": container with ID starting with 3151473df2a102f8fcfc776e69255430c4101bc9f6cf8c2b68442bdbc5e739e8 not found: ID does not exist" Oct 29 22:40:40.389823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.389791 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:40:40.394305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:40.394283 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bef83b-58df7677bd-vhfzf"] Oct 29 22:40:41.276691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:41.276656 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540a0d3b-7030-49ec-bb4e-ba95228a7228" path="/var/lib/kubelet/pods/540a0d3b-7030-49ec-bb4e-ba95228a7228/volumes" Oct 29 22:40:43.676375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:43.676342 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:40:43.676877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:43.676605 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" podUID="0d385160-2163-4699-ae81-0755966873c5" containerName="maas-api" containerID="cri-o://b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f" gracePeriod=30 Oct 29 22:40:44.139953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.139077 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:40:44.217113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.216392 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\") pod \"0d385160-2163-4699-ae81-0755966873c5\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " Oct 29 22:40:44.217113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.216591 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\") pod \"0d385160-2163-4699-ae81-0755966873c5\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " Oct 29 22:40:44.217113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.216688 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\") pod \"0d385160-2163-4699-ae81-0755966873c5\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " Oct 29 22:40:44.217113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.216767 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\" (UniqueName: \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\") pod \"0d385160-2163-4699-ae81-0755966873c5\" (UID: \"0d385160-2163-4699-ae81-0755966873c5\") " Oct 29 22:40:44.219073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.219037 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca" pod "0d385160-2163-4699-ae81-0755966873c5" (UID: "0d385160-2163-4699-ae81-0755966873c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:44.219379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.219354 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf" pod "0d385160-2163-4699-ae81-0755966873c5" (UID: "0d385160-2163-4699-ae81-0755966873c5"). InnerVolumeSpecName "kube-api-access-f4nqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:44.219667 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.219634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs" pod "0d385160-2163-4699-ae81-0755966873c5" (UID: "0d385160-2163-4699-ae81-0755966873c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:44.220452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.220422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls" pod "0d385160-2163-4699-ae81-0755966873c5" (UID: "0d385160-2163-4699-ae81-0755966873c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:44.318986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.318891 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d385160-2163-4699-ae81-0755966873c5-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:44.318986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.318939 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:44.318986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.318958 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d385160-2163-4699-ae81-0755966873c5-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:44.318986 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.318971 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4nqf\" (UniqueName: \"kubernetes.io/projected/0d385160-2163-4699-ae81-0755966873c5-kube-api-access-f4nqf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:44.360274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.360238 4376 generic.go:356] "Generic (PLEG): container finished" podID="0d385160-2163-4699-ae81-0755966873c5" containerID="b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f" exitCode=0 Oct 29 22:40:44.360452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.360288 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" Oct 29 22:40:44.360452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.360324 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" event={"ID":"0d385160-2163-4699-ae81-0755966873c5","Type":"ContainerDied","Data":"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f"} Oct 29 22:40:44.360452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.360381 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz" event={"ID":"0d385160-2163-4699-ae81-0755966873c5","Type":"ContainerDied","Data":"f71ce09f95d6b58be4bb27169a4b3e37c065ccf5347197f4fbcefdbbd6f296dc"} Oct 29 22:40:44.360452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.360408 4376 scope.go:122] "RemoveContainer" containerID="b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f" Oct 29 22:40:44.372617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.372593 4376 scope.go:122] "RemoveContainer" containerID="b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f" Oct 29 22:40:44.372959 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:44.372936 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f\": container with ID starting with b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f not found: ID does not exist" containerID="b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f" Oct 29 22:40:44.373049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.372966 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f"} err="failed to get container status \"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f\": rpc error: code = NotFound desc = could not find container \"b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f\": container with ID starting with b3f339e1cc7b880991eb8451cc178457175f40e5e3f19186e80b7ca6affb571f not found: ID does not exist" Oct 29 22:40:44.419735 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.419693 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:40:44.426837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:44.426804 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0a3d53-559bbcd6f9-z56mz"] Oct 29 22:40:45.277696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:45.277648 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d385160-2163-4699-ae81-0755966873c5" path="/var/lib/kubelet/pods/0d385160-2163-4699-ae81-0755966873c5/volumes" Oct 29 22:40:46.468731 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:46.468694 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:40:47.437238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:47.437195 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:47.437465 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:47.437206 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:48.706065 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:48.706026 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:40:48.706582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:48.706297 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" containerID="cri-o://857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381" gracePeriod=30 Oct 29 22:40:52.491097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:52.491049 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:40:52.491643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:52.491353 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" containerID="cri-o://211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826" gracePeriod=30 Oct 29 22:40:53.009291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:53.009245 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:40:53.009479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:53.009313 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:40:54.139814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.139359 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:40:54.226900 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.226867 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.226946 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227059 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227180 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227209 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227329 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227404 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227437 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.227620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227488 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\") pod \"5f2b1360-5739-4485-b57e-00aa770854a3\" (UID: \"5f2b1360-5739-4485-b57e-00aa770854a3\") " Oct 29 22:40:54.228742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.227198 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.228742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.228577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.229058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.228977 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:54.229208 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.229180 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.229296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.229276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.231959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.231318 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.239120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.239074 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "kube-api-access-cdhvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.239120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.239099 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:54.240373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.240346 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token" pod "5f2b1360-5739-4485-b57e-00aa770854a3" (UID: "5f2b1360-5739-4485-b57e-00aa770854a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.269041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.269008 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:40:54.330487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330451 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330488 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b1360-5739-4485-b57e-00aa770854a3-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330505 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330520 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330532 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2b1360-5739-4485-b57e-00aa770854a3-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330544 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330559 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330572 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2b1360-5739-4485-b57e-00aa770854a3-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.330674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.330583 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdhvm\" (UniqueName: \"kubernetes.io/projected/5f2b1360-5739-4485-b57e-00aa770854a3-kube-api-access-cdhvm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.331810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.331783 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:40:54.431674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.431636 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\" (UniqueName: \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\") pod \"5a19411f-562d-4d77-8a97-e9f73d1079a1\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " Oct 29 22:40:54.431858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.431818 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\") pod \"bb769b61-43aa-4b69-9504-d4a4332cbbed\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " Oct 29 22:40:54.432058 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432039 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\") pod \"5a19411f-562d-4d77-8a97-e9f73d1079a1\" (UID: \"5a19411f-562d-4d77-8a97-e9f73d1079a1\") " Oct 29 22:40:54.432114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432078 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\" (UniqueName: \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\") pod \"bb769b61-43aa-4b69-9504-d4a4332cbbed\" (UID: \"bb769b61-43aa-4b69-9504-d4a4332cbbed\") " Oct 29 22:40:54.432252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432227 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume" pod "bb769b61-43aa-4b69-9504-d4a4332cbbed" (UID: "bb769b61-43aa-4b69-9504-d4a4332cbbed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:54.432624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume" pod "5a19411f-562d-4d77-8a97-e9f73d1079a1" (UID: "5a19411f-562d-4d77-8a97-e9f73d1079a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:54.432775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432757 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19411f-562d-4d77-8a97-e9f73d1079a1-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.432830 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.432776 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb769b61-43aa-4b69-9504-d4a4332cbbed-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.433856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.433838 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7" pod "5a19411f-562d-4d77-8a97-e9f73d1079a1" (UID: "5a19411f-562d-4d77-8a97-e9f73d1079a1"). InnerVolumeSpecName "kube-api-access-j9qs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.434370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.434353 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw" pod "bb769b61-43aa-4b69-9504-d4a4332cbbed" (UID: "bb769b61-43aa-4b69-9504-d4a4332cbbed"). InnerVolumeSpecName "kube-api-access-ll7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.462773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.462738 4376 generic.go:356] "Generic (PLEG): container finished" podID="5f2b1360-5739-4485-b57e-00aa770854a3" containerID="857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381" exitCode=0 Oct 29 22:40:54.463003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.462802 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" Oct 29 22:40:54.463003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.462810 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" event={"ID":"5f2b1360-5739-4485-b57e-00aa770854a3","Type":"ContainerDied","Data":"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381"} Oct 29 22:40:54.463003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.462868 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9" event={"ID":"5f2b1360-5739-4485-b57e-00aa770854a3","Type":"ContainerDied","Data":"f3c3251ef90d8b107509e3b58759a688398f12365c6bf2076f8d0cf6ae634e6a"} Oct 29 22:40:54.463003 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.462891 4376 scope.go:122] "RemoveContainer" containerID="857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381" Oct 29 22:40:54.464615 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.464583 4376 generic.go:356] "Generic (PLEG): container finished" podID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerID="758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3" exitCode=0 Oct 29 22:40:54.464745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.464616 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" event={"ID":"bb769b61-43aa-4b69-9504-d4a4332cbbed","Type":"ContainerDied","Data":"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3"} Oct 29 22:40:54.464745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.464632 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" Oct 29 22:40:54.464745 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.464663 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8" event={"ID":"bb769b61-43aa-4b69-9504-d4a4332cbbed","Type":"ContainerDied","Data":"feb8299a9e341b0de2efabbc5e40d0e6c9251cb4ecadc7526f7ad36ca57172b3"} Oct 29 22:40:54.466339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.466316 4376 generic.go:356] "Generic (PLEG): container finished" podID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerID="7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250" exitCode=0 Oct 29 22:40:54.466339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.466348 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" event={"ID":"5a19411f-562d-4d77-8a97-e9f73d1079a1","Type":"ContainerDied","Data":"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250"} Oct 29 22:40:54.466560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.466369 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" Oct 29 22:40:54.466560 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.466393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm" event={"ID":"5a19411f-562d-4d77-8a97-e9f73d1079a1","Type":"ContainerDied","Data":"31eb0dae10858a7408d5a2a6dc8569367b6ba7a2bd21f26cfd1a83c55ecb3fee"} Oct 29 22:40:54.475520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.475256 4376 scope.go:122] "RemoveContainer" containerID="857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381" Oct 29 22:40:54.475856 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:54.475829 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381\": container with ID starting with 857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381 not found: ID does not exist" containerID="857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381" Oct 29 22:40:54.476369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.475864 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381"} err="failed to get container status \"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381\": rpc error: code = NotFound desc = could not find container \"857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381\": container with ID starting with 857258b880f9cf8cac9511e9ae403228e4a0a7bdb7c066c1fab12a6632644381 not found: ID does not exist" Oct 29 22:40:54.476369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.475881 4376 scope.go:122] "RemoveContainer" containerID="758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3" Oct 29 22:40:54.487765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.487740 4376 scope.go:122] "RemoveContainer" containerID="758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3" Oct 29 22:40:54.488098 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:54.488074 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3\": container with ID starting with 758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3 not found: ID does not exist" containerID="758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3" Oct 29 22:40:54.488190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.488106 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3"} err="failed to get container status \"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3\": rpc error: code = NotFound desc = could not find container \"758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3\": container with ID starting with 758554a288d7cf5ade581191165bcaf5ab79d7427a7e98563f4d6a2e7a1f22f3 not found: ID does not exist" Oct 29 22:40:54.488190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.488125 4376 scope.go:122] "RemoveContainer" containerID="7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250" Oct 29 22:40:54.500328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.500304 4376 scope.go:122] "RemoveContainer" containerID="7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250" Oct 29 22:40:54.500686 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:54.500663 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250\": container with ID starting with 7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250 not found: ID does not exist" containerID="7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250" Oct 29 22:40:54.500770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.500697 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250"} err="failed to get container status \"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250\": rpc error: code = NotFound desc = could not find container \"7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250\": container with ID starting with 7154fa1038942d05a2d48242164b464be0bf41ab67a6148d8747c7c1cc311250 not found: ID does not exist" Oct 29 22:40:54.534532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.534493 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7sw\" (UniqueName: \"kubernetes.io/projected/bb769b61-43aa-4b69-9504-d4a4332cbbed-kube-api-access-ll7sw\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.534532 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.534527 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qs7\" (UniqueName: \"kubernetes.io/projected/5a19411f-562d-4d77-8a97-e9f73d1079a1-kube-api-access-j9qs7\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.553277 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.553237 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:40:54.558636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.558600 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c1015-75b68559f4-vz4s8"] Oct 29 22:40:54.598743 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.598703 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:40:54.603712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.603683 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c1015-868bd8779-5mfsm"] Oct 29 22:40:54.641333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.641300 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:40:54.650434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:54.650397 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-8bzz9"] Oct 29 22:40:55.273558 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:55.273518 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" path="/var/lib/kubelet/pods/5a19411f-562d-4d77-8a97-e9f73d1079a1/volumes" Oct 29 22:40:55.274097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:55.274081 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" path="/var/lib/kubelet/pods/5f2b1360-5739-4485-b57e-00aa770854a3/volumes" Oct 29 22:40:55.274491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:55.274477 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" path="/var/lib/kubelet/pods/bb769b61-43aa-4b69-9504-d4a4332cbbed/volumes" Oct 29 22:40:57.977835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:57.976048 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.047977 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048073 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048146 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048225 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048271 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048303 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048378 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048425 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.048538 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\") pod \"07e732d9-1fa1-43bb-819d-0da8a0b35787\" (UID: \"07e732d9-1fa1-43bb-819d-0da8a0b35787\") " Oct 29 22:40:58.050844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.050533 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:58.051576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.050944 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:58.051576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.050948 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:58.051576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.051078 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:58.051576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.051142 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:58.054962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.054922 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:58.055290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.055255 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:58.058092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.058064 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:58.058245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.058161 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q" pod "07e732d9-1fa1-43bb-819d-0da8a0b35787" (UID: "07e732d9-1fa1-43bb-819d-0da8a0b35787"). InnerVolumeSpecName "kube-api-access-9vl7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151320 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151355 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151366 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151378 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151388 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vl7q\" (UniqueName: \"kubernetes.io/projected/07e732d9-1fa1-43bb-819d-0da8a0b35787-kube-api-access-9vl7q\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151396 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151405 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151413 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e732d9-1fa1-43bb-819d-0da8a0b35787-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.151809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.151421 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e732d9-1fa1-43bb-819d-0da8a0b35787-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:58.538570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.538535 4376 generic.go:356] "Generic (PLEG): container finished" podID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerID="211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826" exitCode=0 Oct 29 22:40:58.538778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.538591 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" Oct 29 22:40:58.538778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.538613 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerDied","Data":"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826"} Oct 29 22:40:58.538778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.538673 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" event={"ID":"07e732d9-1fa1-43bb-819d-0da8a0b35787","Type":"ContainerDied","Data":"0c1ed29f2cbe6ac708404c2db19f5523a5760e11b93d7be3331ec83c3d186788"} Oct 29 22:40:58.538778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.538698 4376 scope.go:122] "RemoveContainer" containerID="211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826" Oct 29 22:40:58.550874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.550850 4376 scope.go:122] "RemoveContainer" containerID="fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e" Oct 29 22:40:58.561686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.561665 4376 scope.go:122] "RemoveContainer" containerID="211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826" Oct 29 22:40:58.562090 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:58.562068 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826\": container with ID starting with 211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826 not found: ID does not exist" containerID="211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826" Oct 29 22:40:58.562157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.562100 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826"} err="failed to get container status \"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826\": rpc error: code = NotFound desc = could not find container \"211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826\": container with ID starting with 211a446af03ef3766957eadcc169992407ede31721e2ae12dfe25f6baa035826 not found: ID does not exist" Oct 29 22:40:58.562157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.562117 4376 scope.go:122] "RemoveContainer" containerID="fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e" Oct 29 22:40:58.562411 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:40:58.562388 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e\": container with ID starting with fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e not found: ID does not exist" containerID="fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e" Oct 29 22:40:58.562461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.562418 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e"} err="failed to get container status \"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e\": rpc error: code = NotFound desc = could not find container \"fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e\": container with ID starting with fb04752ae0efba0b4e2d562c5071b7660b0e4dd835ce15cffdcf5345f394774e not found: ID does not exist" Oct 29 22:40:58.602801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.602768 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:40:58.615343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.615312 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q"] Oct 29 22:40:58.933468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.933370 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:40:58.933468 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:58.933453 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0a3d53-openshift-default-c754858b5-dhf2q" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:59.321770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:40:59.321722 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" path="/var/lib/kubelet/pods/07e732d9-1fa1-43bb-819d-0da8a0b35787/volumes" Oct 29 22:41:05.188121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.188082 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:41:05.189193 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189171 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.189268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189194 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.189268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189215 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" Oct 29 22:41:05.189268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189222 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" Oct 29 22:41:05.189268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189248 4376 cpu_manager.go:414] "containerMap: removing container" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.189268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189256 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189269 4376 cpu_manager.go:414] "containerMap: removing container" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189276 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189298 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189307 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189315 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189322 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189340 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189347 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189358 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189365 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189381 4376 cpu_manager.go:414] "containerMap: removing container" podUID="540a0d3b-7030-49ec-bb4e-ba95228a7228" containerName="maas-api" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189387 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540a0d3b-7030-49ec-bb4e-ba95228a7228" containerName="maas-api" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189399 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189406 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189421 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" Oct 29 22:41:05.189430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189427 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189456 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0d385160-2163-4699-ae81-0755966873c5" containerName="maas-api" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189463 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d385160-2163-4699-ae81-0755966873c5" containerName="maas-api" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189609 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d385160-2163-4699-ae81-0755966873c5" containerName="maas-api" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189622 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189631 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6eb1d1-0e6f-48c7-894f-5e7db11f42ad" containerName="payload-processing" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189642 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189650 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fd685a-3c40-483e-b6fc-5087ec6ce75b" containerName="payload-pre-processing" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189660 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5f92d3-8c87-4b8d-b443-d2d788fb2957" containerName="istio-proxy" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189671 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb769b61-43aa-4b69-9504-d4a4332cbbed" containerName="payload-processing" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189680 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb4b42e-a45d-4576-abb2-629dd96ff906" containerName="payload-pre-processing" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189689 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="540a0d3b-7030-49ec-bb4e-ba95228a7228" containerName="maas-api" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189698 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a19411f-562d-4d77-8a97-e9f73d1079a1" containerName="payload-pre-processing" Oct 29 22:41:05.190039 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.189708 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2b1360-5739-4485-b57e-00aa770854a3" containerName="istio-proxy" Oct 29 22:41:05.195106 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.195081 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.202822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.202791 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:41:05.203809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.203785 4376 cpu_manager.go:414] "containerMap: removing container" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.203809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.203810 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.204037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.203984 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e732d9-1fa1-43bb-819d-0da8a0b35787" containerName="istio-proxy" Oct 29 22:41:05.204037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.204017 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27fa9f4-2bc6-406a-95c3-959ec1d2361d" containerName="istio-proxy" Oct 29 22:41:05.208164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.208144 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.211962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.211938 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-580af2-openshift-default-dockercfg-n7dzr\"" Oct 29 22:41:05.216069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.216043 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:41:05.230378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.230340 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:41:05.302378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302337 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.302594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302482 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.302594 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302557 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.302715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302609 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.302715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302645 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.302715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302665 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.302827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.302867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302841 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cv7h\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.302998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.302974 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.303036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.303098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303085 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.303153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303140 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.303201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303160 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.303201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9d8r\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.303201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.303346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.303406 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303367 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.303466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.303425 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.405346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405312 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.405536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cv7h\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.405536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405448 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.405536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405485 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.405726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405548 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.405726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405632 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.405726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405658 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.405726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405681 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9d8r\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.405726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405710 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405741 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405818 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405855 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405876 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.405946 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406023 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406056 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406111 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406141 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406145 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406174 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406232 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406380 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406487 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.406752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406749 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.407159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406952 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.407159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.406995 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.407159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.407062 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.408610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.408510 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.408738 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.408694 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.409090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.409065 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.409170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.409094 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.430306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.430281 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.430463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.430354 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cv7h\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.430674 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.430656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.430785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.430768 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9d8r\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.514414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.514383 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:05.534292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.534261 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:05.862209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.862156 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:41:05.862939 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:41:05.862877 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084fee8e_781d_417d_a76b_65b26b25c9d1.slice/crio-81a9fb859a3559e45f17be7b1389ebc515d5afc3ff0363a8f4e1211a15a2b7a2 WatchSource:0}: Error finding container 81a9fb859a3559e45f17be7b1389ebc515d5afc3ff0363a8f4e1211a15a2b7a2: Status 404 returned error can't find the container with id 81a9fb859a3559e45f17be7b1389ebc515d5afc3ff0363a8f4e1211a15a2b7a2 Oct 29 22:41:05.865803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.865769 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:05.865869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.865836 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:05.865925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.865876 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:05.866400 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.866344 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:41:05.878283 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:41:05.878257 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1360c02e_3a09_45ca_9d4c_f6241c9c66f0.slice/crio-11198de9e95a7589b4cb60748c9cfed2b41a50cd8ad19d098aa966301d5de926 WatchSource:0}: Error finding container 11198de9e95a7589b4cb60748c9cfed2b41a50cd8ad19d098aa966301d5de926: Status 404 returned error can't find the container with id 11198de9e95a7589b4cb60748c9cfed2b41a50cd8ad19d098aa966301d5de926 Oct 29 22:41:05.881583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.881546 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:05.881694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.881628 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:05.881694 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:05.881667 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:06.629948 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:06.629882 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" event={"ID":"084fee8e-781d-417d-a76b-65b26b25c9d1","Type":"ContainerStarted","Data":"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0"} Oct 29 22:41:06.630388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:06.629969 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" event={"ID":"084fee8e-781d-417d-a76b-65b26b25c9d1","Type":"ContainerStarted","Data":"81a9fb859a3559e45f17be7b1389ebc515d5afc3ff0363a8f4e1211a15a2b7a2"} Oct 29 22:41:06.631778 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:06.631754 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" event={"ID":"1360c02e-3a09-45ca-9d4c-f6241c9c66f0","Type":"ContainerStarted","Data":"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25"} Oct 29 22:41:06.631879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:06.631798 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" event={"ID":"1360c02e-3a09-45ca-9d4c-f6241c9c66f0","Type":"ContainerStarted","Data":"11198de9e95a7589b4cb60748c9cfed2b41a50cd8ad19d098aa966301d5de926"} Oct 29 22:41:06.686851 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:06.686783 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podStartSLOduration=1.686761519 podStartE2EDuration="1.686761519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:06.686542982 +0000 UTC m=+3525.535019829" watchObservedRunningTime="2026-10-29 22:41:06.686761519 +0000 UTC m=+3525.535238368" Oct 29 22:41:07.522445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.520344 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:07.544153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.522688 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:41:07.544153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.522739 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 29 22:41:07.544153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.539281 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:07.544153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.539403 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:41:07.544153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:07.539449 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 29 22:41:08.515273 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:08.515236 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:41:08.515706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:08.515293 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 29 22:41:08.557929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:08.554017 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:41:08.557929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:08.554080 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 29 22:41:09.515078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:09.515040 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:41:09.515274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:09.515103 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 29 22:41:09.556078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:09.556037 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:41:09.556718 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:09.556682 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 29 22:41:10.515305 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:10.515272 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:41:10.515493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:10.515336 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 29 22:41:10.559382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:10.559342 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:41:10.560568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:10.560535 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 29 22:41:11.515018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:11.514974 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.515219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:11.515052 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 29 22:41:11.555362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:11.555323 4376 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.555668 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:11.555636 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 29 22:41:12.520015 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.519984 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:12.520504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.520479 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:12.521154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.521124 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:41:12.540261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.540236 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:12.540482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.540460 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:12.541279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.541264 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:41:12.582005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:12.581944 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podStartSLOduration=7.581925124 podStartE2EDuration="7.581925124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:06.765301563 +0000 UTC m=+3525.613778409" watchObservedRunningTime="2026-10-29 22:41:12.581925124 +0000 UTC m=+3531.430401982" Oct 29 22:41:16.435387 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:41:16.435335 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:41:22.607666 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.607632 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:41:22.608298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.607892 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" containerID="cri-o://ff1996bfd3bee40654da77590fadeb6ababb5f499c0cc452fa9171b7a791e99e" gracePeriod=30 Oct 29 22:41:22.799929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.799023 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:41:22.829452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.829411 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.845941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.841876 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:41:22.870979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.846434 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-90f46c-openshift-default-dockercfg-2gcfd\"" Oct 29 22:41:22.911928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.911877 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.911965 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912024 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912137 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndr2v\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912203 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912292 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912447 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912331 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912484 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:22.912607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:22.912588 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.014780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.014736 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndr2v\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.014806 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.014869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015027 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.014896 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015069 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015172 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015239 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015221 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015266 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015316 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015386 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015614 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015808 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015783 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.015998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.015979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.017402 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.017379 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.017639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.017619 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.057739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.057706 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.057944 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.057787 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndr2v\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.166961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.166854 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:23.498822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.498789 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:41:23.499974 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:41:23.499949 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c5c7b6_bfe1_4a6e_8be5_5429e7a24acd.slice/crio-0b7e10ae58177ffa7dfe8d3cc26bdff42ab4e6a647ab046e98f522f4b6dfeabd WatchSource:0}: Error finding container 0b7e10ae58177ffa7dfe8d3cc26bdff42ab4e6a647ab046e98f522f4b6dfeabd: Status 404 returned error can't find the container with id 0b7e10ae58177ffa7dfe8d3cc26bdff42ab4e6a647ab046e98f522f4b6dfeabd Oct 29 22:41:23.503155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.503115 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:23.503270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.503199 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:23.503270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.503247 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:41:23.915157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.915063 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" event={"ID":"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd","Type":"ContainerStarted","Data":"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707"} Oct 29 22:41:23.915157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.915123 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" event={"ID":"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd","Type":"ContainerStarted","Data":"0b7e10ae58177ffa7dfe8d3cc26bdff42ab4e6a647ab046e98f522f4b6dfeabd"} Oct 29 22:41:23.966395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:23.966330 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podStartSLOduration=1.966312649 podStartE2EDuration="1.966312649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:23.96253498 +0000 UTC m=+3542.811011838" watchObservedRunningTime="2026-10-29 22:41:23.966312649 +0000 UTC m=+3542.814789568" Oct 29 22:41:24.167372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:24.167276 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:24.168929 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:24.168888 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:24.169076 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:24.168961 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:25.167557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:25.167518 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:25.167996 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:25.167577 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:26.167430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:26.167392 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:26.167650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:26.167458 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:26.581116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:26.581076 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:41:26.581300 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:26.581140 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:41:27.167522 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.167488 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:27.167725 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.167547 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:27.817090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.817057 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:41:27.823985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.823876 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.870580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.869603 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27b6db\"/\"default-dockercfg-khq7q\"" Oct 29 22:41:27.870580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.870303 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27b6db\"/\"openshift-service-ca.crt\"" Oct 29 22:41:27.870580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.870408 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27b6db\"/\"kube-root-ca.crt\"" Oct 29 22:41:27.870580 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.870527 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27b6db\"/\"test-model-27b6db-kserve-self-signed-certs\"" Oct 29 22:41:27.895307 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.895266 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:41:27.925343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.925297 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.925688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.925662 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.925846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.925826 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.926094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.926073 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.926264 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.926246 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.926425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.926407 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dzb\" (UniqueName: \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:27.978491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.978460 4376 generic.go:356] "Generic (PLEG): container finished" podID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerID="ff1996bfd3bee40654da77590fadeb6ababb5f499c0cc452fa9171b7a791e99e" exitCode=0 Oct 29 22:41:27.978792 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.978773 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerDied","Data":"ff1996bfd3bee40654da77590fadeb6ababb5f499c0cc452fa9171b7a791e99e"} Oct 29 22:41:27.979022 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:27.979003 4376 scope.go:122] "RemoveContainer" containerID="ae732de450232978466bb7246d639597d4fa6f841abb5bc4fe9e5dbcc4371666" Oct 29 22:41:28.028608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028577 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.028765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028674 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.028765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.028879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.028879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028807 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.028879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.028839 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dzb\" (UniqueName: \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.029130 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.029102 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.029220 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.029194 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.029282 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.029265 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.031019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.030997 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.031478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.031460 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.122937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.119981 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dzb\" (UniqueName: \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\") pod \"test-model-27b6db-kserve-f8d98977c-p9sl5\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.176052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.174273 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:28.176052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.174343 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:28.192941 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.190117 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:41:28.202384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.202344 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233128 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233181 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233233 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233341 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233378 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233598 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233633 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.233693 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\") pod \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\" (UID: \"f7c191fe-b715-4094-9b8d-3bd281efd9b8\") " Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.235502 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.235698 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.236146 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.236426 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:28.240930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.236754 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:28.263964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.244080 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:28.263964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.244374 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:28.263964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.261155 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "kube-api-access-c727g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:28.263964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.261284 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy" pod "f7c191fe-b715-4094-9b8d-3bd281efd9b8" (UID: "f7c191fe-b715-4094-9b8d-3bd281efd9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364344 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364377 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c727g\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-kube-api-access-c727g\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364389 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364400 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364408 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364416 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364424 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364432 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c191fe-b715-4094-9b8d-3bd281efd9b8-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.364926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.364440 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c191fe-b715-4094-9b8d-3bd281efd9b8-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:28.578404 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.578370 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:41:28.578626 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:41:28.578602 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec83798f_31a3_4d39_8de9_16f11610491f.slice/crio-e6f3e157e6b42a4d06ae930b833c32a665c1f32193c3215775a07d9594bb1fd0 WatchSource:0}: Error finding container e6f3e157e6b42a4d06ae930b833c32a665c1f32193c3215775a07d9594bb1fd0: Status 404 returned error can't find the container with id e6f3e157e6b42a4d06ae930b833c32a665c1f32193c3215775a07d9594bb1fd0 Oct 29 22:41:28.988398 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.988361 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" event={"ID":"ec83798f-31a3-4d39-8de9-16f11610491f","Type":"ContainerStarted","Data":"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e"} Oct 29 22:41:28.988597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.988414 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" event={"ID":"ec83798f-31a3-4d39-8de9-16f11610491f","Type":"ContainerStarted","Data":"e6f3e157e6b42a4d06ae930b833c32a665c1f32193c3215775a07d9594bb1fd0"} Oct 29 22:41:28.989968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.989944 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" Oct 29 22:41:28.990096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.989999 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb" event={"ID":"f7c191fe-b715-4094-9b8d-3bd281efd9b8","Type":"ContainerDied","Data":"6b09f2d8ba61ea4ef0f26eae35589dc8cc50bd17618a5d70eb62025560eaec6d"} Oct 29 22:41:28.990096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:28.990059 4376 scope.go:122] "RemoveContainer" containerID="ff1996bfd3bee40654da77590fadeb6ababb5f499c0cc452fa9171b7a791e99e" Oct 29 22:41:29.052958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.050746 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" podStartSLOduration=2.050722477 podStartE2EDuration="2.050722477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:29.031284692 +0000 UTC m=+3547.879761537" watchObservedRunningTime="2026-10-29 22:41:29.050722477 +0000 UTC m=+3547.899199324" Oct 29 22:41:29.079716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.079681 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:41:29.086471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.086437 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c1015-openshift-default-5664fd5d69-n99bb"] Oct 29 22:41:29.170708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.169611 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 22:41:29.170708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.169679 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 22:41:29.203250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.203216 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:29.275055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:29.274972 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" path="/var/lib/kubelet/pods/f7c191fe-b715-4094-9b8d-3bd281efd9b8/volumes" Oct 29 22:41:30.174483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:30.174448 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:30.174991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:30.174950 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:30.175583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:30.175553 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:41:31.850587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.850542 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:41:31.851434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.851412 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:41:31.851434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.851430 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:41:31.851608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.851554 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:41:31.851608 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.851579 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:41:31.856713 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.856692 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.876449 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.876416 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:41:31.882934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.882748 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-580af2\"/\"default-dockercfg-4hjdh\"" Oct 29 22:41:31.882934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.882859 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-580af2\"/\"test-model-580af2-kserve-self-signed-certs\"" Oct 29 22:41:31.889215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.889173 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-580af2\"/\"kube-root-ca.crt\"" Oct 29 22:41:31.889349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.889270 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-580af2\"/\"openshift-service-ca.crt\"" Oct 29 22:41:31.945107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945069 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.945411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945376 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.945547 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945430 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.945617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945577 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.945801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5km4\" (UniqueName: \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:31.945922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:31.945885 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047259 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047314 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047345 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047391 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047566 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5km4\" (UniqueName: \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047670 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047877 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.047969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.047867 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.048152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.048127 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.049981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.049956 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.050126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.050069 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.073274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.073246 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5km4\" (UniqueName: \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\") pod \"test-model-580af2-kserve-768c9f789f-n8n74\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.190483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.190389 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:32.732128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:32.731794 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:41:32.733962 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:41:32.733256 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa00f6c_e1bd_41b7_96fe_cacd076832d8.slice/crio-4c6fab10a5e53a81792445e0e8b0238c9871191f5e872b2414e00ee24a3adbc1 WatchSource:0}: Error finding container 4c6fab10a5e53a81792445e0e8b0238c9871191f5e872b2414e00ee24a3adbc1: Status 404 returned error can't find the container with id 4c6fab10a5e53a81792445e0e8b0238c9871191f5e872b2414e00ee24a3adbc1 Oct 29 22:41:33.038577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:33.038491 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" event={"ID":"afa00f6c-e1bd-41b7-96fe-cacd076832d8","Type":"ContainerStarted","Data":"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31"} Oct 29 22:41:33.038577 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:33.038534 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" event={"ID":"afa00f6c-e1bd-41b7-96fe-cacd076832d8","Type":"ContainerStarted","Data":"4c6fab10a5e53a81792445e0e8b0238c9871191f5e872b2414e00ee24a3adbc1"} Oct 29 22:41:33.082343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:33.082287 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" podStartSLOduration=2.082272868 podStartE2EDuration="2.082272868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:33.081319136 +0000 UTC m=+3551.929795983" watchObservedRunningTime="2026-10-29 22:41:33.082272868 +0000 UTC m=+3551.930749713" Oct 29 22:41:33.210927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:33.207929 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:37.589768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:37.589679 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 22:41:37.590708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:37.590658 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" podUID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" containerName="maas-api" containerID="cri-o://354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e" gracePeriod=30 Oct 29 22:41:38.061926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.057827 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 22:41:38.110276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.110212 4376 generic.go:356] "Generic (PLEG): container finished" podID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" containerID="354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e" exitCode=0 Oct 29 22:41:38.110505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.110297 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" Oct 29 22:41:38.110505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.110329 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" event={"ID":"a63ec822-782a-41ee-b6d0-81ae8a852fc3","Type":"ContainerDied","Data":"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e"} Oct 29 22:41:38.110505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.110393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc" event={"ID":"a63ec822-782a-41ee-b6d0-81ae8a852fc3","Type":"ContainerDied","Data":"1da9394e0ec333161142e99f77d87a1aaedd845ef81ec665e0e0f4684f0bd7cb"} Oct 29 22:41:38.110649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.110488 4376 scope.go:122] "RemoveContainer" containerID="354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e" Oct 29 22:41:38.127731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.127702 4376 scope.go:122] "RemoveContainer" containerID="354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e" Oct 29 22:41:38.128104 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:41:38.128073 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e\": container with ID starting with 354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e not found: ID does not exist" containerID="354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e" Oct 29 22:41:38.128184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.128114 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e"} err="failed to get container status \"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e\": rpc error: code = NotFound desc = could not find container \"354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e\": container with ID starting with 354d665a5824515eaffd3c57254d6d2b51601a25b4b01eec09c93a63a81dbe4e not found: ID does not exist" Oct 29 22:41:38.156732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.156703 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\") pod \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " Oct 29 22:41:38.156880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.156838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\") pod \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " Oct 29 22:41:38.156998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.156973 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\" (UniqueName: \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\") pod \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " Oct 29 22:41:38.157230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.157206 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\") pod \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\" (UID: \"a63ec822-782a-41ee-b6d0-81ae8a852fc3\") " Oct 29 22:41:38.157673 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.157644 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca" pod "a63ec822-782a-41ee-b6d0-81ae8a852fc3" (UID: "a63ec822-782a-41ee-b6d0-81ae8a852fc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:38.158038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.158012 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63ec822-782a-41ee-b6d0-81ae8a852fc3-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:38.159187 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.159165 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls" pod "a63ec822-782a-41ee-b6d0-81ae8a852fc3" (UID: "a63ec822-782a-41ee-b6d0-81ae8a852fc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:38.159324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.159302 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs" pod "a63ec822-782a-41ee-b6d0-81ae8a852fc3" (UID: "a63ec822-782a-41ee-b6d0-81ae8a852fc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:38.159966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.159949 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb" pod "a63ec822-782a-41ee-b6d0-81ae8a852fc3" (UID: "a63ec822-782a-41ee-b6d0-81ae8a852fc3"). InnerVolumeSpecName "kube-api-access-mnhxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:38.203257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.203230 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:38.215770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.215743 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:38.259945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.259882 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:38.259945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.259941 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhxb\" (UniqueName: \"kubernetes.io/projected/a63ec822-782a-41ee-b6d0-81ae8a852fc3-kube-api-access-mnhxb\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:38.260175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.259958 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63ec822-782a-41ee-b6d0-81ae8a852fc3-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:38.461777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.461704 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 22:41:38.467484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:38.466985 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-92d984-856648dd77-5jmqc"] Oct 29 22:41:39.131952 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:39.131918 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:41:39.274010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:39.273977 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" path="/var/lib/kubelet/pods/a63ec822-782a-41ee-b6d0-81ae8a852fc3/volumes" Oct 29 22:41:42.190732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:42.190694 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:42.205771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:42.205744 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:43.164102 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:43.164071 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:41:46.447647 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:41:46.447604 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:41:46.452520 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:46.452489 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 22:41:46.452769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:46.452726 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" containerID="cri-o://b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22" gracePeriod=30 Oct 29 22:41:49.147567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:49.147522 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:41:49.148000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:49.147594 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:41:51.841982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.841955 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 22:41:51.957567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.957499 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.958722 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.957893 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.959177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.958932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.959177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.958976 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.959177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.959021 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.960217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.959531 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.960217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.959630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.960217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.959656 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.960217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.959717 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\") pod \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\" (UID: \"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f\") " Oct 29 22:41:51.961844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.961637 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:51.961844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.961649 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:51.962032 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.961844 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:51.962192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.962145 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:51.963776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.963732 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:51.963895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.963774 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:51.967290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.967268 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:51.967409 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.967383 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "kube-api-access-j5jr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:51.973563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:51.973533 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy" pod "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" (UID: "9fd5fc22-a3d3-4c94-88f5-fea689a73e6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.062870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062786 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.062870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062825 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.062870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062842 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.062870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062854 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.062870 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062867 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jr4\" (UniqueName: \"kubernetes.io/projected/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-kube-api-access-j5jr4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.063171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062879 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.063171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062893 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.063171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062922 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.063171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.062936 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.239481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.239442 4376 generic.go:356] "Generic (PLEG): container finished" podID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerID="b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22" exitCode=0 Oct 29 22:41:52.239650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.239491 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" event={"ID":"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f","Type":"ContainerDied","Data":"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22"} Oct 29 22:41:52.239650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.239557 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" event={"ID":"9fd5fc22-a3d3-4c94-88f5-fea689a73e6f","Type":"ContainerDied","Data":"785f50b6124092ad0e26d002974eb3ea585299f39381429b8f99f3e1299534cd"} Oct 29 22:41:52.239650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.239492 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89" Oct 29 22:41:52.239650 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.239584 4376 scope.go:122] "RemoveContainer" containerID="b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22" Oct 29 22:41:52.251803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.251774 4376 scope.go:122] "RemoveContainer" containerID="b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22" Oct 29 22:41:52.252112 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:41:52.252088 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22\": container with ID starting with b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22 not found: ID does not exist" containerID="b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22" Oct 29 22:41:52.252194 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.252120 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22"} err="failed to get container status \"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22\": rpc error: code = NotFound desc = could not find container \"b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22\": container with ID starting with b1f54156025f07ac6ce7b31f41b6e811fce0cd534371f804b6c3a0f5c9037d22 not found: ID does not exist" Oct 29 22:41:52.305139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.305060 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 22:41:52.317433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:52.317363 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-6fk89"] Oct 29 22:41:53.275028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:53.274996 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" path="/var/lib/kubelet/pods/9fd5fc22-a3d3-4c94-88f5-fea689a73e6f/volumes" Oct 29 22:41:54.661048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:54.660963 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 22:41:56.998840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:41:56.998806 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 22:42:09.683508 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:09.683466 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" containerID="cri-o://d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f" gracePeriod=45 Oct 29 22:42:09.982858 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:09.982831 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:42:10.111029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.110992 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.111219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111070 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.111329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111306 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.111470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111439 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.111549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111515 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.111617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111590 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\" (UniqueName: \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.111675 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.111627 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\") pod \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\" (UID: \"4698b55f-414e-42d3-a92f-ad7ecfeec77e\") " Oct 29 22:42:10.112369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.112339 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.112519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.112498 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.112648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.112562 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.112648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.112584 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.114227 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.114183 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:10.114429 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.114413 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.114895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.114870 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd" pod "4698b55f-414e-42d3-a92f-ad7ecfeec77e" (UID: "4698b55f-414e-42d3-a92f-ad7ecfeec77e"). InnerVolumeSpecName "kube-api-access-9whtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:10.213736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.213698 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9whtd\" (UniqueName: \"kubernetes.io/projected/4698b55f-414e-42d3-a92f-ad7ecfeec77e-kube-api-access-9whtd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.213736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.213728 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.213736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.213737 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4698b55f-414e-42d3-a92f-ad7ecfeec77e-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.213736 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.213745 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4698b55f-414e-42d3-a92f-ad7ecfeec77e-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.378479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.378444 4376 generic.go:356] "Generic (PLEG): container finished" podID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerID="d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f" exitCode=0 Oct 29 22:42:10.378680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.378498 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" Oct 29 22:42:10.378680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.378522 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" event={"ID":"4698b55f-414e-42d3-a92f-ad7ecfeec77e","Type":"ContainerDied","Data":"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f"} Oct 29 22:42:10.378680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.378568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl" event={"ID":"4698b55f-414e-42d3-a92f-ad7ecfeec77e","Type":"ContainerDied","Data":"8dc01474b06f2fdd8298909d931bf21579bf23ecfdb6a731f547877656ab9c89"} Oct 29 22:42:10.378680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.378589 4376 scope.go:122] "RemoveContainer" containerID="d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f" Oct 29 22:42:10.394732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.394708 4376 scope.go:122] "RemoveContainer" containerID="d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f" Oct 29 22:42:10.395090 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:10.395065 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f\": container with ID starting with d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f not found: ID does not exist" containerID="d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f" Oct 29 22:42:10.395188 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.395101 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f"} err="failed to get container status \"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f\": rpc error: code = NotFound desc = could not find container \"d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f\": container with ID starting with d9746b7b327e9d22ce1886bf16d65be48509fc7df290d91a085e0114f0441e0f not found: ID does not exist" Oct 29 22:42:10.435078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.435045 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 22:42:10.447572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:10.447542 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-2-w1-29c2f2-kserve-7d68598b55-tw2kl"] Oct 29 22:42:11.276059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:11.276026 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" path="/var/lib/kubelet/pods/4698b55f-414e-42d3-a92f-ad7ecfeec77e/volumes" Oct 29 22:42:12.020811 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.020775 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" containerID="cri-o://a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a" gracePeriod=45 Oct 29 22:42:12.309505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.309475 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:42:12.399252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.399217 4376 generic.go:356] "Generic (PLEG): container finished" podID="90f0d211-5adf-40ca-8c85-56878719bf15" containerID="a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a" exitCode=0 Oct 29 22:42:12.399423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.399274 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" Oct 29 22:42:12.399423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.399300 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" event={"ID":"90f0d211-5adf-40ca-8c85-56878719bf15","Type":"ContainerDied","Data":"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a"} Oct 29 22:42:12.399423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.399362 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7" event={"ID":"90f0d211-5adf-40ca-8c85-56878719bf15","Type":"ContainerDied","Data":"dbf7ae115a61149252bc04fb1a264d30861dd0ec59ca6e77fde42caa62a8224e"} Oct 29 22:42:12.399423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.399387 4376 scope.go:122] "RemoveContainer" containerID="a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a" Oct 29 22:42:12.413844 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.413818 4376 scope.go:122] "RemoveContainer" containerID="a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a" Oct 29 22:42:12.414248 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:12.414218 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a\": container with ID starting with a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a not found: ID does not exist" containerID="a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a" Oct 29 22:42:12.414354 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.414262 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a"} err="failed to get container status \"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a\": rpc error: code = NotFound desc = could not find container \"a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a\": container with ID starting with a6212a8a57d979fe641b8042b66907dc99bc3764c8f46af0552cea70acbd591a not found: ID does not exist" Oct 29 22:42:12.447524 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.447478 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.447706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.447629 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.447706 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.447680 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\" (UniqueName: \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.448007 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.447981 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.448134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.448134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448099 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\" (UniqueName: \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.448258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448127 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.448258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448153 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\") pod \"90f0d211-5adf-40ca-8c85-56878719bf15\" (UID: \"90f0d211-5adf-40ca-8c85-56878719bf15\") " Oct 29 22:42:12.448865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448837 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.448995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.448870 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.449116 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.449097 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.451148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.451123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:12.451259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.451154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "kube-api-access-cp2z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:12.451259 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.451234 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm" pod "90f0d211-5adf-40ca-8c85-56878719bf15" (UID: "90f0d211-5adf-40ca-8c85-56878719bf15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.550422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.550342 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.550422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.550370 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp2z6\" (UniqueName: \"kubernetes.io/projected/90f0d211-5adf-40ca-8c85-56878719bf15-kube-api-access-cp2z6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.550422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.550380 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f0d211-5adf-40ca-8c85-56878719bf15-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.550422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.550398 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f0d211-5adf-40ca-8c85-56878719bf15-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.750170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.750135 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 22:42:12.762557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.762524 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-2-w4-cea94a-kserve-bf45ff8cf-8n8n7"] Oct 29 22:42:12.837974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.837860 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:42:12.838840 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:12.838797 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" podUID="4fc313ec-8b15-48da-9467-c70f2282594f" containerName="maas-api" containerID="cri-o://aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45" gracePeriod=30 Oct 29 22:42:13.250557 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.250198 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:42:13.317094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.315196 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" path="/var/lib/kubelet/pods/90f0d211-5adf-40ca-8c85-56878719bf15/volumes" Oct 29 22:42:13.374145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.374107 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\" (UniqueName: \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\") pod \"4fc313ec-8b15-48da-9467-c70f2282594f\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " Oct 29 22:42:13.374459 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.374435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\") pod \"4fc313ec-8b15-48da-9467-c70f2282594f\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " Oct 29 22:42:13.374548 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.374490 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\") pod \"4fc313ec-8b15-48da-9467-c70f2282594f\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " Oct 29 22:42:13.374548 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.374536 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\") pod \"4fc313ec-8b15-48da-9467-c70f2282594f\" (UID: \"4fc313ec-8b15-48da-9467-c70f2282594f\") " Oct 29 22:42:13.375059 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.375030 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca" pod "4fc313ec-8b15-48da-9467-c70f2282594f" (UID: "4fc313ec-8b15-48da-9467-c70f2282594f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:13.375320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.375293 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc313ec-8b15-48da-9467-c70f2282594f-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:13.377677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.377648 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs" pod "4fc313ec-8b15-48da-9467-c70f2282594f" (UID: "4fc313ec-8b15-48da-9467-c70f2282594f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:13.377783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.377650 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb" pod "4fc313ec-8b15-48da-9467-c70f2282594f" (UID: "4fc313ec-8b15-48da-9467-c70f2282594f"). InnerVolumeSpecName "kube-api-access-d7tcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:13.377783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.377762 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls" pod "4fc313ec-8b15-48da-9467-c70f2282594f" (UID: "4fc313ec-8b15-48da-9467-c70f2282594f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:13.408394 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.408364 4376 generic.go:356] "Generic (PLEG): container finished" podID="4fc313ec-8b15-48da-9467-c70f2282594f" containerID="aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45" exitCode=0 Oct 29 22:42:13.408504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.408418 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" Oct 29 22:42:13.408504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.408444 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" event={"ID":"4fc313ec-8b15-48da-9467-c70f2282594f","Type":"ContainerDied","Data":"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45"} Oct 29 22:42:13.408504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.408496 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8" event={"ID":"4fc313ec-8b15-48da-9467-c70f2282594f","Type":"ContainerDied","Data":"5f161452bc73df21bd41b04a5817fd1d59b9bcdfd0fa8ece6b8aa0333970deca"} Oct 29 22:42:13.408659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.408519 4376 scope.go:122] "RemoveContainer" containerID="aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45" Oct 29 22:42:13.425949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.425925 4376 scope.go:122] "RemoveContainer" containerID="aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45" Oct 29 22:42:13.426285 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:13.426260 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45\": container with ID starting with aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45 not found: ID does not exist" containerID="aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45" Oct 29 22:42:13.426375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.426296 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45"} err="failed to get container status \"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45\": rpc error: code = NotFound desc = could not find container \"aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45\": container with ID starting with aba7661c7f0ff511b286efe1e3cfea7ce5c41bc680c11827ad90ba9ef9199d45 not found: ID does not exist" Oct 29 22:42:13.467253 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.467225 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:42:13.470642 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.470622 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-99e420-8589ddccdd-nqkr8"] Oct 29 22:42:13.477020 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.476995 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7tcb\" (UniqueName: \"kubernetes.io/projected/4fc313ec-8b15-48da-9467-c70f2282594f-kube-api-access-d7tcb\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:13.477123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.477022 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:13.477123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:13.477038 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fc313ec-8b15-48da-9467-c70f2282594f-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:15.273311 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:15.273277 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc313ec-8b15-48da-9467-c70f2282594f" path="/var/lib/kubelet/pods/4fc313ec-8b15-48da-9467-c70f2282594f/volumes" Oct 29 22:42:16.435488 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:16.435450 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:42:46.457260 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:46.457220 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:42:50.019417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:50.019382 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:42:55.019720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:55.019677 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" containerID="cri-o://71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92" gracePeriod=25 Oct 29 22:42:56.093699 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:56.093658 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:57.086079 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.086042 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:42:57.086571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.086513 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" podUID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" containerName="maas-api" containerID="cri-o://ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca" gracePeriod=30 Oct 29 22:42:57.459946 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.459899 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:42:57.566617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.566577 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\") pod \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " Oct 29 22:42:57.567501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.567474 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\") pod \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " Oct 29 22:42:57.568165 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.568043 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca" pod "5fd25ab2-2855-46a2-aed0-8b4abfad235e" (UID: "5fd25ab2-2855-46a2-aed0-8b4abfad235e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:57.568418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.568401 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\" (UniqueName: \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\") pod \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " Oct 29 22:42:57.568963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.568942 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\") pod \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\" (UID: \"5fd25ab2-2855-46a2-aed0-8b4abfad235e\") " Oct 29 22:42:57.590176 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.569800 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd25ab2-2855-46a2-aed0-8b4abfad235e-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.590688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.572600 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls" pod "5fd25ab2-2855-46a2-aed0-8b4abfad235e" (UID: "5fd25ab2-2855-46a2-aed0-8b4abfad235e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:57.590835 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.572727 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg" pod "5fd25ab2-2855-46a2-aed0-8b4abfad235e" (UID: "5fd25ab2-2855-46a2-aed0-8b4abfad235e"). InnerVolumeSpecName "kube-api-access-qbrqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:57.590979 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.576055 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs" pod "5fd25ab2-2855-46a2-aed0-8b4abfad235e" (UID: "5fd25ab2-2855-46a2-aed0-8b4abfad235e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:57.692554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.692467 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrqg\" (UniqueName: \"kubernetes.io/projected/5fd25ab2-2855-46a2-aed0-8b4abfad235e-kube-api-access-qbrqg\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.692554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.692504 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.692554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.692517 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd25ab2-2855-46a2-aed0-8b4abfad235e-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:57.780082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.780039 4376 generic.go:356] "Generic (PLEG): container finished" podID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" containerID="ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca" exitCode=0 Oct 29 22:42:57.780082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.780077 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" Oct 29 22:42:57.780320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.780102 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" event={"ID":"5fd25ab2-2855-46a2-aed0-8b4abfad235e","Type":"ContainerDied","Data":"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca"} Oct 29 22:42:57.780320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.780162 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27" event={"ID":"5fd25ab2-2855-46a2-aed0-8b4abfad235e","Type":"ContainerDied","Data":"a3589090be9aeadc68eb901d9fb0e413685cce677e88f03c199d88e16d89e1a1"} Oct 29 22:42:57.780320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.780187 4376 scope.go:122] "RemoveContainer" containerID="ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca" Oct 29 22:42:57.792440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.792422 4376 scope.go:122] "RemoveContainer" containerID="ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca" Oct 29 22:42:57.792690 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:42:57.792669 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca\": container with ID starting with ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca not found: ID does not exist" containerID="ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca" Oct 29 22:42:57.792749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.792701 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca"} err="failed to get container status \"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca\": rpc error: code = NotFound desc = could not find container \"ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca\": container with ID starting with ba9823ceda7d63057be582f7b4c2982befdf881198659e95e262c88acc97c2ca not found: ID does not exist" Oct 29 22:42:57.848395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.848360 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:42:57.851777 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:57.851751 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-15154b-d89dcfdb-nng27"] Oct 29 22:42:59.273634 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:42:59.273599 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" path="/var/lib/kubelet/pods/5fd25ab2-2855-46a2-aed0-8b4abfad235e/volumes" Oct 29 22:43:06.093023 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:06.092940 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:06.216768 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:06.216732 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:43:06.217376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:06.216968 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" containerID="cri-o://a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3" gracePeriod=30 Oct 29 22:43:06.243838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:06.243800 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:43:06.244068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:06.243881 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:43:11.494535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.494466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:43:11.530646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.530611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.530837 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.530656 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.530893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.530837 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.530893 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.530872 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531009 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.530965 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531060 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531022 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531124 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531112 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531156 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531189 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531173 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.531302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531220 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\") pod \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\" (UID: \"51fd011b-f743-4c5e-b61a-a3c62d5ed463\") " Oct 29 22:43:11.531359 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531310 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:11.531704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531653 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.531704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.531704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531682 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.532094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.531968 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.532094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.532004 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.532094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.532016 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.532094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.532029 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.532094 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.532041 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.533410 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.533380 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:11.533638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.533606 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:11.533739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.533647 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:11.533739 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.533694 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb" pod "51fd011b-f743-4c5e-b61a-a3c62d5ed463" (UID: "51fd011b-f743-4c5e-b61a-a3c62d5ed463"). InnerVolumeSpecName "kube-api-access-th7zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:11.634331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.634291 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.634331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.634326 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.634331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.634335 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7zb\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-kube-api-access-th7zb\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.634587 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.634345 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fd011b-f743-4c5e-b61a-a3c62d5ed463-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.783755 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.783663 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 22:43:11.921004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.920954 4376 generic.go:356] "Generic (PLEG): container finished" podID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerID="a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3" exitCode=0 Oct 29 22:43:11.921218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.921012 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" Oct 29 22:43:11.921218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.921044 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerDied","Data":"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3"} Oct 29 22:43:11.921218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.921117 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6" event={"ID":"51fd011b-f743-4c5e-b61a-a3c62d5ed463","Type":"ContainerDied","Data":"66d6ff3c7110c516e24a1c810da2e2cd1b6d821c44058c82a4938b16f814ce72"} Oct 29 22:43:11.921218 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.921142 4376 scope.go:122] "RemoveContainer" containerID="a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3" Oct 29 22:43:11.933325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.933296 4376 scope.go:122] "RemoveContainer" containerID="cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49" Oct 29 22:43:11.944155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.944128 4376 scope.go:122] "RemoveContainer" containerID="a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3" Oct 29 22:43:11.944477 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:11.944459 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3\": container with ID starting with a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3 not found: ID does not exist" containerID="a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3" Oct 29 22:43:11.944529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.944488 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3"} err="failed to get container status \"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3\": rpc error: code = NotFound desc = could not find container \"a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3\": container with ID starting with a09d1abd5853b5fed4d95ca5bcf198902387cf13dc3f79463b7ef369d750c0c3 not found: ID does not exist" Oct 29 22:43:11.944529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.944505 4376 scope.go:122] "RemoveContainer" containerID="cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49" Oct 29 22:43:11.944780 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:11.944758 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49\": container with ID starting with cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49 not found: ID does not exist" containerID="cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49" Oct 29 22:43:11.944833 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.944788 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49"} err="failed to get container status \"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49\": rpc error: code = NotFound desc = could not find container \"cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49\": container with ID starting with cb9f9330406bf1bfe1067d2408f60598e0ff24ea2808ee123411a809f1e05f49 not found: ID does not exist" Oct 29 22:43:11.985503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.985464 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:43:11.988810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:11.988783 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-15154b-openshift-default-cdf946659-cjzv6"] Oct 29 22:43:13.273215 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:13.273182 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" path="/var/lib/kubelet/pods/51fd011b-f743-4c5e-b61a-a3c62d5ed463/volumes" Oct 29 22:43:15.302540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.302507 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:43:15.398576 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.398539 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\" (UniqueName: \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\") pod \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " Oct 29 22:43:15.398891 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.398874 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\") pod \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\" (UID: \"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059\") " Oct 29 22:43:15.399296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.399266 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume" pod "ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" (UID: "ef1e3fd0-9ba4-4d71-904a-1ef68df3b059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:15.399448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.399428 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:15.400710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.400680 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc" pod "ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" (UID: "ef1e3fd0-9ba4-4d71-904a-1ef68df3b059"). InnerVolumeSpecName "kube-api-access-xdlbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:15.500443 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.500401 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdlbc\" (UniqueName: \"kubernetes.io/projected/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059-kube-api-access-xdlbc\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:15.950653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.950615 4376 generic.go:356] "Generic (PLEG): container finished" podID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerID="71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92" exitCode=0 Oct 29 22:43:15.950653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.950657 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" event={"ID":"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059","Type":"ContainerDied","Data":"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92"} Oct 29 22:43:15.950963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.950676 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" Oct 29 22:43:15.950963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.950699 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47" event={"ID":"ef1e3fd0-9ba4-4d71-904a-1ef68df3b059","Type":"ContainerDied","Data":"c38d6265b22e4a1fcd9ca917c2a1f8e794509353ab80c84bbce9c41cceae8024"} Oct 29 22:43:15.950963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.950720 4376 scope.go:122] "RemoveContainer" containerID="71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92" Oct 29 22:43:15.961943 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.961886 4376 scope.go:122] "RemoveContainer" containerID="71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92" Oct 29 22:43:15.962254 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:15.962233 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92\": container with ID starting with 71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92 not found: ID does not exist" containerID="71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92" Oct 29 22:43:15.962302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:15.962263 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92"} err="failed to get container status \"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92\": rpc error: code = NotFound desc = could not find container \"71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92\": container with ID starting with 71b389889af15e28611dfaedee4024b1e36f12a5f82854305663b86572c76a92 not found: ID does not exist" Oct 29 22:43:16.009013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.008925 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:43:16.020638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.020599 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c06a2c-7fcccf9667-42p47"] Oct 29 22:43:16.063152 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.063113 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 22:43:16.063401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.063371 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" podUID="842bac1d-a464-4937-8668-72bb7966e6e0" containerName="maas-api" containerID="cri-o://eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1" gracePeriod=30 Oct 29 22:43:16.416527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.416497 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 22:43:16.484941 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:16.482070 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:43:16.523219 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.523117 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\") pod \"842bac1d-a464-4937-8668-72bb7966e6e0\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " Oct 29 22:43:16.523412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.523317 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\") pod \"842bac1d-a464-4937-8668-72bb7966e6e0\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " Oct 29 22:43:16.523412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.523386 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\" (UniqueName: \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\") pod \"842bac1d-a464-4937-8668-72bb7966e6e0\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " Oct 29 22:43:16.523537 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.523415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\") pod \"842bac1d-a464-4937-8668-72bb7966e6e0\" (UID: \"842bac1d-a464-4937-8668-72bb7966e6e0\") " Oct 29 22:43:16.524041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.524012 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca" pod "842bac1d-a464-4937-8668-72bb7966e6e0" (UID: "842bac1d-a464-4937-8668-72bb7966e6e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:16.524201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.524157 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/842bac1d-a464-4937-8668-72bb7966e6e0-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:16.526563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.526536 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs" pod "842bac1d-a464-4937-8668-72bb7966e6e0" (UID: "842bac1d-a464-4937-8668-72bb7966e6e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:16.526965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.526916 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls" pod "842bac1d-a464-4937-8668-72bb7966e6e0" (UID: "842bac1d-a464-4937-8668-72bb7966e6e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:16.529112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.529092 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr" pod "842bac1d-a464-4937-8668-72bb7966e6e0" (UID: "842bac1d-a464-4937-8668-72bb7966e6e0"). InnerVolumeSpecName "kube-api-access-6bhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:16.625090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.625049 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhtr\" (UniqueName: \"kubernetes.io/projected/842bac1d-a464-4937-8668-72bb7966e6e0-kube-api-access-6bhtr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:16.625090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.625082 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:16.625090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.625095 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/842bac1d-a464-4937-8668-72bb7966e6e0-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:16.966018 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.965978 4376 generic.go:356] "Generic (PLEG): container finished" podID="842bac1d-a464-4937-8668-72bb7966e6e0" containerID="eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1" exitCode=0 Oct 29 22:43:16.966209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.966030 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" Oct 29 22:43:16.966209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.966069 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" event={"ID":"842bac1d-a464-4937-8668-72bb7966e6e0","Type":"ContainerDied","Data":"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1"} Oct 29 22:43:16.966209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.966153 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4" event={"ID":"842bac1d-a464-4937-8668-72bb7966e6e0","Type":"ContainerDied","Data":"c2343ae96b71143f0821b5bf488d4e17940bec326cb2f0362b740db15d453f36"} Oct 29 22:43:16.966349 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.966211 4376 scope.go:122] "RemoveContainer" containerID="eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1" Oct 29 22:43:16.980983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.980956 4376 scope.go:122] "RemoveContainer" containerID="eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1" Oct 29 22:43:16.982893 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:16.982865 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1\": container with ID starting with eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1 not found: ID does not exist" containerID="eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1" Oct 29 22:43:16.983033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:16.982897 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1"} err="failed to get container status \"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1\": rpc error: code = NotFound desc = could not find container \"eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1\": container with ID starting with eeeeec413e4b6a5a4ad4e8481f2e1901e0e445b3652094826995014098e0cdf1 not found: ID does not exist" Oct 29 22:43:17.028621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:17.028588 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 22:43:17.041274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:17.041240 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29c2f2-f696fcc78-2v9c4"] Oct 29 22:43:17.273869 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:17.273783 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842bac1d-a464-4937-8668-72bb7966e6e0" path="/var/lib/kubelet/pods/842bac1d-a464-4937-8668-72bb7966e6e0/volumes" Oct 29 22:43:17.274326 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:17.274308 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" path="/var/lib/kubelet/pods/ef1e3fd0-9ba4-4d71-904a-1ef68df3b059/volumes" Oct 29 22:43:18.343926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.339384 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 22:43:18.343926 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.339619 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" podUID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" containerName="maas-api" containerID="cri-o://0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c" gracePeriod=30 Oct 29 22:43:18.769100 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.769066 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 22:43:18.882049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.878397 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\" (UniqueName: \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\") pod \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " Oct 29 22:43:18.882049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.878459 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\") pod \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " Oct 29 22:43:18.882049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.878690 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\") pod \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " Oct 29 22:43:18.882049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.878839 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\") pod \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\" (UID: \"5fb09b2e-a108-4f20-8404-a3d06e01e1c4\") " Oct 29 22:43:18.885825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.882896 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca" pod "5fb09b2e-a108-4f20-8404-a3d06e01e1c4" (UID: "5fb09b2e-a108-4f20-8404-a3d06e01e1c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:18.901677 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.901640 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls" pod "5fb09b2e-a108-4f20-8404-a3d06e01e1c4" (UID: "5fb09b2e-a108-4f20-8404-a3d06e01e1c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:18.933320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.925120 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks" pod "5fb09b2e-a108-4f20-8404-a3d06e01e1c4" (UID: "5fb09b2e-a108-4f20-8404-a3d06e01e1c4"). InnerVolumeSpecName "kube-api-access-s6vks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:18.947316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.947234 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs" pod "5fb09b2e-a108-4f20-8404-a3d06e01e1c4" (UID: "5fb09b2e-a108-4f20-8404-a3d06e01e1c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:18.981044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.981010 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:18.981044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.981041 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:18.981044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.981051 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vks\" (UniqueName: \"kubernetes.io/projected/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-kube-api-access-s6vks\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:18.981331 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.981060 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb09b2e-a108-4f20-8404-a3d06e01e1c4-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:18.982470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.982439 4376 generic.go:356] "Generic (PLEG): container finished" podID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" containerID="0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c" exitCode=0 Oct 29 22:43:18.982586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.982497 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" Oct 29 22:43:18.982586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.982498 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" event={"ID":"5fb09b2e-a108-4f20-8404-a3d06e01e1c4","Type":"ContainerDied","Data":"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c"} Oct 29 22:43:18.982658 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.982633 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg" event={"ID":"5fb09b2e-a108-4f20-8404-a3d06e01e1c4","Type":"ContainerDied","Data":"c870c4d38558f62aaef7bf1d9bdaec28efc2cb186186256272ea73d2246ffd8e"} Oct 29 22:43:18.982696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.982657 4376 scope.go:122] "RemoveContainer" containerID="0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c" Oct 29 22:43:18.993274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.993204 4376 scope.go:122] "RemoveContainer" containerID="0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c" Oct 29 22:43:18.993526 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:18.993506 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c\": container with ID starting with 0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c not found: ID does not exist" containerID="0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c" Oct 29 22:43:18.993582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:18.993536 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c"} err="failed to get container status \"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c\": rpc error: code = NotFound desc = could not find container \"0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c\": container with ID starting with 0124719aa2289db95cbc58b75eed8de06634fc51ac3dfb3512f1fd224642a08c not found: ID does not exist" Oct 29 22:43:19.039276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:19.039242 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 22:43:19.047271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:19.047239 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cea94a-b465f7c98-d5lcg"] Oct 29 22:43:19.277611 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:19.277577 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" path="/var/lib/kubelet/pods/5fb09b2e-a108-4f20-8404-a3d06e01e1c4/volumes" Oct 29 22:43:25.324128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:25.324029 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 22:43:25.324637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:25.324337 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" containerID="cri-o://93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31" gracePeriod=30 Oct 29 22:43:25.555696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:25.555654 4376 scope.go:122] "RemoveContainer" containerID="b41fc16f407bd1da84c2de9f5bae2cc5d37b6779975220dc39658fe02ee36535" Oct 29 22:43:26.806186 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:26.806127 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" containerID="cri-o://2d6203ae1fe152e0cb9aa0db23797fdff065af4531b548bd5187422f398ffbf5" gracePeriod=45 Oct 29 22:43:27.046155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.046092 4376 generic.go:356] "Generic (PLEG): container finished" podID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerID="2d6203ae1fe152e0cb9aa0db23797fdff065af4531b548bd5187422f398ffbf5" exitCode=0 Oct 29 22:43:27.046437 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.046238 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" event={"ID":"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3","Type":"ContainerDied","Data":"2d6203ae1fe152e0cb9aa0db23797fdff065af4531b548bd5187422f398ffbf5"} Oct 29 22:43:27.108992 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.108942 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:43:27.213780 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.213748 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\" (UniqueName: \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.213838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.213985 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214035 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214148 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214268 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.214356 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214340 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\") pod \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\" (UID: \"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3\") " Oct 29 22:43:27.214435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214318 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.214502 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.214404 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.215111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.215088 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.215111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.215115 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.215295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.215129 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.216367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.216334 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.216500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.216412 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:27.216500 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.216422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j" pod "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" (UID: "d19e4df8-c3f6-4acf-95b1-62dfcb501bf3"). InnerVolumeSpecName "kube-api-access-78p6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:27.316330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.316239 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.316330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.316267 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.316330 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.316279 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p6j\" (UniqueName: \"kubernetes.io/projected/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3-kube-api-access-78p6j\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.957221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.957179 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 22:43:27.957725 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:27.957448 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" containerID="cri-o://207b6d44919378a4018d87a5fdb68bebab80696c7e7a42badc15e41d482005fa" gracePeriod=30 Oct 29 22:43:28.057091 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:28.057051 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" Oct 29 22:43:28.057328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:28.057050 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw" event={"ID":"d19e4df8-c3f6-4acf-95b1-62dfcb501bf3","Type":"ContainerDied","Data":"9586b5400664d4a777c3270535a364ddd53d6c7844a36b6ab603b386becc8b14"} Oct 29 22:43:28.057328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:28.057191 4376 scope.go:122] "RemoveContainer" containerID="2d6203ae1fe152e0cb9aa0db23797fdff065af4531b548bd5187422f398ffbf5" Oct 29 22:43:28.117752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:28.117709 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 22:43:28.127185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:28.127147 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-w3-82db63-kserve-85fdcb446f-vxxcw"] Oct 29 22:43:29.273515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:29.273484 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" path="/var/lib/kubelet/pods/d19e4df8-c3f6-4acf-95b1-62dfcb501bf3/volumes" Oct 29 22:43:30.630789 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.630759 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 22:43:30.774784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.774744 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.774822 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.774845 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.774882 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775048 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775015 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775054 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775186 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775228 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775229 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.775302 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775255 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\") pod \"f6c04004-8e34-43fd-bce0-cf58df2383f4\" (UID: \"f6c04004-8e34-43fd-bce0-cf58df2383f4\") " Oct 29 22:43:30.775573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775261 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:30.775573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775460 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.775573 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775479 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.775785 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.775725 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.776274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.776250 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.776382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.776277 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c04004-8e34-43fd-bce0-cf58df2383f4-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.776382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.776291 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.776382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.776303 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.776382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.776316 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.777630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.777604 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.777725 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.777634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:30.777772 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.777760 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:30.777806 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.777781 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9" pod "f6c04004-8e34-43fd-bce0-cf58df2383f4" (UID: "f6c04004-8e34-43fd-bce0-cf58df2383f4"). InnerVolumeSpecName "kube-api-access-r7pb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:30.878567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.878420 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.878567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.878467 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-kube-api-access-r7pb9\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.878567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.878479 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.878567 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:30.878491 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c04004-8e34-43fd-bce0-cf58df2383f4-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:31.090477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.090440 4376 generic.go:356] "Generic (PLEG): container finished" podID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerID="93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31" exitCode=0 Oct 29 22:43:31.090719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.090490 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" Oct 29 22:43:31.090719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.090535 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" event={"ID":"f6c04004-8e34-43fd-bce0-cf58df2383f4","Type":"ContainerDied","Data":"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31"} Oct 29 22:43:31.090719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.090591 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz" event={"ID":"f6c04004-8e34-43fd-bce0-cf58df2383f4","Type":"ContainerDied","Data":"af66117f3878f1955698d18cfe06e4de611745d7ba83e103155ff726c6f329cc"} Oct 29 22:43:31.090719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.090620 4376 scope.go:122] "RemoveContainer" containerID="93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31" Oct 29 22:43:31.102382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.102351 4376 scope.go:122] "RemoveContainer" containerID="93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31" Oct 29 22:43:31.102717 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:31.102695 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31\": container with ID starting with 93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31 not found: ID does not exist" containerID="93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31" Oct 29 22:43:31.102823 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.102725 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31"} err="failed to get container status \"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31\": rpc error: code = NotFound desc = could not find container \"93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31\": container with ID starting with 93d531f7188f2c3db2454a8964505553c3f0b50e6c55dc8ba19c0c1d75b6ab31 not found: ID does not exist" Oct 29 22:43:31.153252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.153145 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 22:43:31.165860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.165818 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-7m8hz"] Oct 29 22:43:31.273340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:31.273304 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" path="/var/lib/kubelet/pods/f6c04004-8e34-43fd-bce0-cf58df2383f4/volumes" Oct 29 22:43:33.106871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.106838 4376 generic.go:356] "Generic (PLEG): container finished" podID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerID="207b6d44919378a4018d87a5fdb68bebab80696c7e7a42badc15e41d482005fa" exitCode=0 Oct 29 22:43:33.107301 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.106943 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" event={"ID":"fa6d09c5-b567-4547-b8e3-075959fccbe0","Type":"ContainerDied","Data":"207b6d44919378a4018d87a5fdb68bebab80696c7e7a42badc15e41d482005fa"} Oct 29 22:43:33.237266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.237195 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 22:43:33.420274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420236 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420321 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420378 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420494 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420409 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420513 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420531 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420579 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420689 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420643 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.420942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420714 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.420942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420736 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:33.420942 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420753 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\") pod \"fa6d09c5-b567-4547-b8e3-075959fccbe0\" (UID: \"fa6d09c5-b567-4547-b8e3-075959fccbe0\") " Oct 29 22:43:33.421133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.420962 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.421191 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.421291 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421263 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.421589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421561 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.421589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421584 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.421773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421598 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa6d09c5-b567-4547-b8e3-075959fccbe0-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.421773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421609 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.421773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.421620 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.423201 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.423176 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:33.423292 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.423259 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:33.423483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.423464 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.423483 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.423475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s" pod "fa6d09c5-b567-4547-b8e3-075959fccbe0" (UID: "fa6d09c5-b567-4547-b8e3-075959fccbe0"). InnerVolumeSpecName "kube-api-access-2sb8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:33.523343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.523246 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb8s\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-kube-api-access-2sb8s\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.523343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.523286 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.523343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.523300 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.523343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:33.523312 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa6d09c5-b567-4547-b8e3-075959fccbe0-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:34.048945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.048898 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:43:34.083749 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.083709 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:43:34.114938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.114782 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" Oct 29 22:43:34.114938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.114792 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn" event={"ID":"fa6d09c5-b567-4547-b8e3-075959fccbe0","Type":"ContainerDied","Data":"7d21a229c4b533a381693d7619304326d990766b2bf14e25868cdc6c1636254b"} Oct 29 22:43:34.114938 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.114871 4376 scope.go:122] "RemoveContainer" containerID="207b6d44919378a4018d87a5fdb68bebab80696c7e7a42badc15e41d482005fa" Oct 29 22:43:34.174190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.174158 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 22:43:34.182529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:34.182496 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-gdgvn"] Oct 29 22:43:35.272776 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:35.272736 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" path="/var/lib/kubelet/pods/fa6d09c5-b567-4547-b8e3-075959fccbe0/volumes" Oct 29 22:43:39.050011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:39.049963 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" containerID="cri-o://6801cd90e55913e82aa9fea753141071874c2cbfc7d01ba5e8a03e1211ff03d9" gracePeriod=25 Oct 29 22:43:39.084092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:39.084043 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" containerID="cri-o://916dd924fef9b93a7f19d73f063260258ee91e93461d9ef925311896c4f34235" gracePeriod=25 Oct 29 22:43:41.517198 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:41.517150 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:41.517607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:41.517195 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:46.435102 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:43:46.435060 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:43:51.518134 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:51.518090 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:51.518651 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:51.518155 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:59.311370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.310657 4376 generic.go:356] "Generic (PLEG): container finished" podID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerID="916dd924fef9b93a7f19d73f063260258ee91e93461d9ef925311896c4f34235" exitCode=0 Oct 29 22:43:59.311370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.310800 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" event={"ID":"63ec2e5a-2fc1-434a-a023-24016a6333d0","Type":"ContainerDied","Data":"916dd924fef9b93a7f19d73f063260258ee91e93461d9ef925311896c4f34235"} Oct 29 22:43:59.312661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.312638 4376 generic.go:356] "Generic (PLEG): container finished" podID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerID="6801cd90e55913e82aa9fea753141071874c2cbfc7d01ba5e8a03e1211ff03d9" exitCode=0 Oct 29 22:43:59.312876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.312861 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" event={"ID":"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699","Type":"ContainerDied","Data":"6801cd90e55913e82aa9fea753141071874c2cbfc7d01ba5e8a03e1211ff03d9"} Oct 29 22:43:59.376403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.376376 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:43:59.413609 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.413586 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:43:59.427338 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427313 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\") pod \"63ec2e5a-2fc1-434a-a023-24016a6333d0\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " Oct 29 22:43:59.427475 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427432 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\" (UniqueName: \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\") pod \"63ec2e5a-2fc1-434a-a023-24016a6333d0\" (UID: \"63ec2e5a-2fc1-434a-a023-24016a6333d0\") " Oct 29 22:43:59.427535 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427486 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\") pod \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " Oct 29 22:43:59.427606 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427590 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\" (UniqueName: \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\") pod \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\" (UID: \"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699\") " Oct 29 22:43:59.427661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427636 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume" pod "63ec2e5a-2fc1-434a-a023-24016a6333d0" (UID: "63ec2e5a-2fc1-434a-a023-24016a6333d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:59.427953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.427926 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume" pod "9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" (UID: "9ea6b72b-dd47-4ce5-af72-3b3f9b90f699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:59.428175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.428157 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.428252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.428178 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ec2e5a-2fc1-434a-a023-24016a6333d0-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.429678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.429653 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g" pod "9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" (UID: "9ea6b72b-dd47-4ce5-af72-3b3f9b90f699"). InnerVolumeSpecName "kube-api-access-k2p7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:59.429964 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.429937 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g" pod "63ec2e5a-2fc1-434a-a023-24016a6333d0" (UID: "63ec2e5a-2fc1-434a-a023-24016a6333d0"). InnerVolumeSpecName "kube-api-access-x8g9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:59.529497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.529458 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2p7g\" (UniqueName: \"kubernetes.io/projected/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699-kube-api-access-k2p7g\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.529497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:43:59.529493 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8g9g\" (UniqueName: \"kubernetes.io/projected/63ec2e5a-2fc1-434a-a023-24016a6333d0-kube-api-access-x8g9g\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:00.081981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.081950 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:44:00.082161 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.082142 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" containerID="cri-o://e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707" gracePeriod=30 Oct 29 22:44:00.177251 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.177212 4376 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:44:00.177627 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.177595 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:44:00.333723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.333631 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" event={"ID":"63ec2e5a-2fc1-434a-a023-24016a6333d0","Type":"ContainerDied","Data":"2a15e1da2d8e1a4605515ae85676c5c33be8cc847be47280db1c0adb4164d10c"} Oct 29 22:44:00.333723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.333711 4376 scope.go:122] "RemoveContainer" containerID="916dd924fef9b93a7f19d73f063260258ee91e93461d9ef925311896c4f34235" Oct 29 22:44:00.334252 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.333875 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd" Oct 29 22:44:00.350963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.348106 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" event={"ID":"9ea6b72b-dd47-4ce5-af72-3b3f9b90f699","Type":"ContainerDied","Data":"93aab8924da339c4357b97fc90753a766c6f5ef359b52cdca73793d71efb1cc7"} Oct 29 22:44:00.350963 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.348210 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz" Oct 29 22:44:00.362550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.362519 4376 scope.go:122] "RemoveContainer" containerID="6801cd90e55913e82aa9fea753141071874c2cbfc7d01ba5e8a03e1211ff03d9" Oct 29 22:44:00.401602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.401569 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:44:00.407626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.407597 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90f46c-6474f4c4f5-pkktd"] Oct 29 22:44:00.461503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.461462 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:44:00.466766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:00.466737 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90f46c-75f9b5ffcd-lzssz"] Oct 29 22:44:01.273279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:01.273247 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" path="/var/lib/kubelet/pods/63ec2e5a-2fc1-434a-a023-24016a6333d0/volumes" Oct 29 22:44:01.273653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:01.273639 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" path="/var/lib/kubelet/pods/9ea6b72b-dd47-4ce5-af72-3b3f9b90f699/volumes" Oct 29 22:44:05.351945 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.351858 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:44:05.387403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.387371 4376 generic.go:356] "Generic (PLEG): container finished" podID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerID="e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707" exitCode=0 Oct 29 22:44:05.387586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.387418 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" Oct 29 22:44:05.387586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.387454 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" event={"ID":"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd","Type":"ContainerDied","Data":"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707"} Oct 29 22:44:05.387586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.387510 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp" event={"ID":"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd","Type":"ContainerDied","Data":"0b7e10ae58177ffa7dfe8d3cc26bdff42ab4e6a647ab046e98f522f4b6dfeabd"} Oct 29 22:44:05.387586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.387531 4376 scope.go:122] "RemoveContainer" containerID="e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707" Oct 29 22:44:05.398072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.398053 4376 scope.go:122] "RemoveContainer" containerID="e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707" Oct 29 22:44:05.398420 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:44:05.398397 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707\": container with ID starting with e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707 not found: ID does not exist" containerID="e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707" Oct 29 22:44:05.398503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.398431 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707"} err="failed to get container status \"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707\": rpc error: code = NotFound desc = could not find container \"e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707\": container with ID starting with e0cedc32a8a65d771195bd908ff20eca08df4edd8c32d38d2642d50b0193d707 not found: ID does not exist" Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.411730 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.411785 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.411816 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.411929 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412019 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412066 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412261 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412552 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412308 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\") pod \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\" (UID: \"92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd\") " Oct 29 22:44:05.412746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412719 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.412951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412898 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:05.412951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.412939 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.413414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.413099 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.413414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.413123 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.413414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.413137 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.413414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.413154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.414036 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.414011 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.415747 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.415720 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:05.415847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.415772 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "kube-api-access-ndr2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:05.416249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.416229 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.416595 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.416564 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo" pod "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" (UID: "92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:05.514672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514629 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.514672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514664 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.514672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514674 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.514951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514683 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.514951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514693 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.514951 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.514701 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndr2v\" (UniqueName: \"kubernetes.io/projected/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd-kube-api-access-ndr2v\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.740096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.740055 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:44:05.754373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:05.754320 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-wk9kp"] Oct 29 22:44:07.272763 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:07.272727 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" path="/var/lib/kubelet/pods/92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd/volumes" Oct 29 22:44:16.434743 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:44:16.434697 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:44:23.138828 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.138786 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 22:44:23.139514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.139044 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" podUID="34b5ca49-5e59-48df-b0a7-838f91298c62" containerName="maas-api" containerID="cri-o://e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59" gracePeriod=30 Oct 29 22:44:23.467726 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.467697 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 22:44:23.504145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.504103 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") pod \"34b5ca49-5e59-48df-b0a7-838f91298c62\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " Oct 29 22:44:23.504368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.504227 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\") pod \"34b5ca49-5e59-48df-b0a7-838f91298c62\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " Oct 29 22:44:23.504652 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.504260 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\") pod \"34b5ca49-5e59-48df-b0a7-838f91298c62\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " Oct 29 22:44:23.505064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.504843 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca" pod "34b5ca49-5e59-48df-b0a7-838f91298c62" (UID: "34b5ca49-5e59-48df-b0a7-838f91298c62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:23.505271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.505254 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\" (UniqueName: \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\") pod \"34b5ca49-5e59-48df-b0a7-838f91298c62\" (UID: \"34b5ca49-5e59-48df-b0a7-838f91298c62\") " Oct 29 22:44:23.506515 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.506492 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b5ca49-5e59-48df-b0a7-838f91298c62-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:23.507262 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.507236 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls" pod "34b5ca49-5e59-48df-b0a7-838f91298c62" (UID: "34b5ca49-5e59-48df-b0a7-838f91298c62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:23.509343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.509300 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb" pod "34b5ca49-5e59-48df-b0a7-838f91298c62" (UID: "34b5ca49-5e59-48df-b0a7-838f91298c62"). InnerVolumeSpecName "kube-api-access-p9lnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:23.534899 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.534855 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs" pod "34b5ca49-5e59-48df-b0a7-838f91298c62" (UID: "34b5ca49-5e59-48df-b0a7-838f91298c62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:23.567566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.567521 4376 generic.go:356] "Generic (PLEG): container finished" podID="34b5ca49-5e59-48df-b0a7-838f91298c62" containerID="e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59" exitCode=0 Oct 29 22:44:23.567767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.567687 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" event={"ID":"34b5ca49-5e59-48df-b0a7-838f91298c62","Type":"ContainerDied","Data":"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59"} Oct 29 22:44:23.567767 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.567743 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" event={"ID":"34b5ca49-5e59-48df-b0a7-838f91298c62","Type":"ContainerDied","Data":"f3f2ca888680033b03efacdfc6cb6c7b2148d798865850a6b802d3371b6def5c"} Oct 29 22:44:23.567934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.567769 4376 scope.go:122] "RemoveContainer" containerID="e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59" Oct 29 22:44:23.568034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.568015 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n" Oct 29 22:44:23.598436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.598405 4376 scope.go:122] "RemoveContainer" containerID="e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59" Oct 29 22:44:23.603141 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:44:23.603099 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59\": container with ID starting with e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59 not found: ID does not exist" containerID="e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59" Oct 29 22:44:23.603365 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.603330 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59"} err="failed to get container status \"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59\": rpc error: code = NotFound desc = could not find container \"e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59\": container with ID starting with e602960f0bcf922da91f8248b1ac7e7619f6691722110c6f8e5bd4dd88852b59 not found: ID does not exist" Oct 29 22:44:23.608499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.608472 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:23.608499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.608500 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b5ca49-5e59-48df-b0a7-838f91298c62-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:23.608691 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.608515 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lnb\" (UniqueName: \"kubernetes.io/projected/34b5ca49-5e59-48df-b0a7-838f91298c62-kube-api-access-p9lnb\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:23.628959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.628922 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 22:44:23.636315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:23.636280 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82db63-66cc565d4c-qdl5n"] Oct 29 22:44:25.273825 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:25.273787 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b5ca49-5e59-48df-b0a7-838f91298c62" path="/var/lib/kubelet/pods/34b5ca49-5e59-48df-b0a7-838f91298c62/volumes" Oct 29 22:44:25.606824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:25.606737 4376 scope.go:122] "RemoveContainer" containerID="015d3c21332e889968ee17a3dc779f73328a9b3dabe421548d13809e80a28234" Oct 29 22:44:25.616427 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:44:25.616404 4376 scope.go:122] "RemoveContainer" containerID="1b13a98f337ca69b9d592a6886f1d4f1ad29bccbacc48828aba54e8b8b52abcf" Oct 29 22:44:46.434121 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:44:46.434033 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:45:05.186411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.186373 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:45:05.187185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.186668 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" podUID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" containerName="maas-api" containerID="cri-o://02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8" gracePeriod=30 Oct 29 22:45:05.524570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.524542 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:45:05.559064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.556653 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\") pod \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " Oct 29 22:45:05.559064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.556789 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") pod \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " Oct 29 22:45:05.559064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.556836 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\") pod \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " Oct 29 22:45:05.559064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.557022 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\" (UniqueName: \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\") pod \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\" (UID: \"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9\") " Oct 29 22:45:05.559507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.559261 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca" pod "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" (UID: "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:05.588162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.587383 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls" pod "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" (UID: "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:05.588162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.587560 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m" pod "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" (UID: "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9"). InnerVolumeSpecName "kube-api-access-8dh9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:05.616966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.604362 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs" pod "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" (UID: "c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:05.659304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.659171 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dh9m\" (UniqueName: \"kubernetes.io/projected/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-kube-api-access-8dh9m\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:05.659304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.659239 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:05.659304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.659257 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:05.659304 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.659268 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:05.850230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.850145 4376 generic.go:356] "Generic (PLEG): container finished" podID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" containerID="02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8" exitCode=0 Oct 29 22:45:05.850230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.850192 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" Oct 29 22:45:05.850438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.850225 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" event={"ID":"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9","Type":"ContainerDied","Data":"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8"} Oct 29 22:45:05.850438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.850285 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg" event={"ID":"c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9","Type":"ContainerDied","Data":"94529efdda21b9092d044861cc034f33c7cee854b17cb6c318785df9c4a2bc68"} Oct 29 22:45:05.850438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.850315 4376 scope.go:122] "RemoveContainer" containerID="02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8" Oct 29 22:45:05.862129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.862107 4376 scope.go:122] "RemoveContainer" containerID="02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8" Oct 29 22:45:05.862405 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:05.862383 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8\": container with ID starting with 02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8 not found: ID does not exist" containerID="02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8" Oct 29 22:45:05.862455 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.862414 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8"} err="failed to get container status \"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8\": rpc error: code = NotFound desc = could not find container \"02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8\": container with ID starting with 02a884e7225410d2753d3950091df18d9d0d278fa7f308fe9033d60f8cd882d8 not found: ID does not exist" Oct 29 22:45:05.907762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.907728 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:45:05.914142 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:05.914114 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d8760-64785d86b9-cqbxg"] Oct 29 22:45:07.272773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:07.272728 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" path="/var/lib/kubelet/pods/c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9/volumes" Oct 29 22:45:16.434757 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:16.434719 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:45:19.588534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:19.588503 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:45:29.456753 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:29.456710 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:45:29.457229 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:29.456963 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" containerID="cri-o://8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914" gracePeriod=30 Oct 29 22:45:30.338353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:30.338312 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:45:30.375954 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:30.375922 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:45:34.611163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.611089 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" podUID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" containerName="main" containerID="cri-o://7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f" gracePeriod=45 Oct 29 22:45:34.745586 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.745559 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:45:34.821653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821325 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.821653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821390 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.821653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821471 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.821653 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821581 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.822068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821810 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.822068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.821992 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.822068 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.822054 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.822272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.822088 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.822272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.822125 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\") pod \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\" (UID: \"411dd6f0-fe08-41b2-982d-d2cdfe258fad\") " Oct 29 22:45:34.823355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.822965 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:34.823355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.823055 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:34.823355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.823291 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:34.823643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.823458 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:34.823731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.823706 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:34.826392 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.826356 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:34.831975 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.831947 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "kube-api-access-6sn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:34.833270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.833241 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:34.834182 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.834159 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy" pod "411dd6f0-fe08-41b2-982d-d2cdfe258fad" (UID: "411dd6f0-fe08-41b2-982d-d2cdfe258fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:34.870013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.869987 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:45:34.925464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925393 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.925464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925431 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.925464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925446 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.925464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925460 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.925464 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925474 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.926050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925488 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn9z\" (UniqueName: \"kubernetes.io/projected/411dd6f0-fe08-41b2-982d-d2cdfe258fad-kube-api-access-6sn9z\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.926050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925502 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.926050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925518 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.926050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:34.925533 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411dd6f0-fe08-41b2-982d-d2cdfe258fad-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.026064 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.025972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026066 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026145 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\" (UniqueName: \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026190 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026221 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026296 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:35.026501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026325 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:35.026501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\") pod \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\" (UID: \"c354ed1c-da1b-459f-b7b1-4089adb4d83d\") " Oct 29 22:45:35.026766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.026565 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:35.027362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.027333 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.027470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.027366 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.027470 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.027381 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.029051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.029028 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:35.029051 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.029036 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:35.029209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.029069 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk" pod "c354ed1c-da1b-459f-b7b1-4089adb4d83d" (UID: "c354ed1c-da1b-459f-b7b1-4089adb4d83d"). InnerVolumeSpecName "kube-api-access-9bznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:35.070775 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.070733 4376 generic.go:356] "Generic (PLEG): container finished" podID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerID="8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914" exitCode=0 Oct 29 22:45:35.071002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.070789 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" Oct 29 22:45:35.071002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.070822 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" event={"ID":"411dd6f0-fe08-41b2-982d-d2cdfe258fad","Type":"ContainerDied","Data":"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914"} Oct 29 22:45:35.071002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.070893 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk" event={"ID":"411dd6f0-fe08-41b2-982d-d2cdfe258fad","Type":"ContainerDied","Data":"0d8d1822282cdd13c42eb24b6ddcf3809478689ae9d50282c59ba1b3abd18055"} Oct 29 22:45:35.071002 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.070931 4376 scope.go:122] "RemoveContainer" containerID="8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914" Oct 29 22:45:35.072434 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.072405 4376 generic.go:356] "Generic (PLEG): container finished" podID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" containerID="7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f" exitCode=0 Oct 29 22:45:35.072536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.072438 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" Oct 29 22:45:35.072536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.072476 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" event={"ID":"c354ed1c-da1b-459f-b7b1-4089adb4d83d","Type":"ContainerDied","Data":"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f"} Oct 29 22:45:35.072536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.072509 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f" event={"ID":"c354ed1c-da1b-459f-b7b1-4089adb4d83d","Type":"ContainerDied","Data":"a18d2c15329d6c7c707cd67021f9d54421982620348fd19428f29cf15f38fa01"} Oct 29 22:45:35.082474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.082425 4376 scope.go:122] "RemoveContainer" containerID="8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914" Oct 29 22:45:35.082750 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:35.082726 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914\": container with ID starting with 8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914 not found: ID does not exist" containerID="8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914" Oct 29 22:45:35.082824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.082758 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914"} err="failed to get container status \"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914\": rpc error: code = NotFound desc = could not find container \"8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914\": container with ID starting with 8648c2f79b45bc471ce8709a9bcc83ed0148d93f901f80ce1077077e5d8e2914 not found: ID does not exist" Oct 29 22:45:35.082824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.082776 4376 scope.go:122] "RemoveContainer" containerID="7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f" Oct 29 22:45:35.093968 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.093896 4376 scope.go:122] "RemoveContainer" containerID="7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f" Oct 29 22:45:35.094227 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:35.094207 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f\": container with ID starting with 7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f not found: ID does not exist" containerID="7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f" Oct 29 22:45:35.094285 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.094236 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f"} err="failed to get container status \"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f\": rpc error: code = NotFound desc = could not find container \"7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f\": container with ID starting with 7c9cf0f5081070b38632474de513a62b305618a57a9f4f6480d2109d31fc451f not found: ID does not exist" Oct 29 22:45:35.127177 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.127140 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:45:35.128892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.128867 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bznk\" (UniqueName: \"kubernetes.io/projected/c354ed1c-da1b-459f-b7b1-4089adb4d83d-kube-api-access-9bznk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.128892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.128891 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c354ed1c-da1b-459f-b7b1-4089adb4d83d-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.129103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.128932 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c354ed1c-da1b-459f-b7b1-4089adb4d83d-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:35.132551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.132527 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-563f39/test-model-563f39-kserve-dfbc666cf-9kt8f"] Oct 29 22:45:35.171411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.171365 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:45:35.176751 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.176723 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-t8tkk"] Oct 29 22:45:35.272308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.272264 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" path="/var/lib/kubelet/pods/411dd6f0-fe08-41b2-982d-d2cdfe258fad/volumes" Oct 29 22:45:35.272865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.272841 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" path="/var/lib/kubelet/pods/c354ed1c-da1b-459f-b7b1-4089adb4d83d/volumes" Oct 29 22:45:35.338696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.338586 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" containerID="cri-o://0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4" gracePeriod=25 Oct 29 22:45:35.376276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:35.376230 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" containerID="cri-o://e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad" gracePeriod=25 Oct 29 22:45:38.920995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:38.883970 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:45:38.920995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:38.884284 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" podUID="c239f9a9-c47e-440c-874f-ad512120b4f2" containerName="maas-api" containerID="cri-o://c96f6a1e529b1a366a876cab64d13247962dd3076743edb027a318470352194d" gracePeriod=30 Oct 29 22:45:39.118419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.118381 4376 generic.go:356] "Generic (PLEG): container finished" podID="c239f9a9-c47e-440c-874f-ad512120b4f2" containerID="c96f6a1e529b1a366a876cab64d13247962dd3076743edb027a318470352194d" exitCode=0 Oct 29 22:45:39.118623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.118450 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" event={"ID":"c239f9a9-c47e-440c-874f-ad512120b4f2","Type":"ContainerDied","Data":"c96f6a1e529b1a366a876cab64d13247962dd3076743edb027a318470352194d"} Oct 29 22:45:39.205630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.205599 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:45:39.292403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.292349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\") pod \"c239f9a9-c47e-440c-874f-ad512120b4f2\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " Oct 29 22:45:39.297287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.297257 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\" (UniqueName: \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\") pod \"c239f9a9-c47e-440c-874f-ad512120b4f2\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " Oct 29 22:45:39.297621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.297599 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") pod \"c239f9a9-c47e-440c-874f-ad512120b4f2\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " Oct 29 22:45:39.297761 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.297745 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\") pod \"c239f9a9-c47e-440c-874f-ad512120b4f2\" (UID: \"c239f9a9-c47e-440c-874f-ad512120b4f2\") " Oct 29 22:45:39.299435 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.298521 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca" pod "c239f9a9-c47e-440c-874f-ad512120b4f2" (UID: "c239f9a9-c47e-440c-874f-ad512120b4f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:39.362370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.362324 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls" pod "c239f9a9-c47e-440c-874f-ad512120b4f2" (UID: "c239f9a9-c47e-440c-874f-ad512120b4f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:39.362734 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.362709 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz" pod "c239f9a9-c47e-440c-874f-ad512120b4f2" (UID: "c239f9a9-c47e-440c-874f-ad512120b4f2"). InnerVolumeSpecName "kube-api-access-2r8gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:39.368967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.368896 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs" pod "c239f9a9-c47e-440c-874f-ad512120b4f2" (UID: "c239f9a9-c47e-440c-874f-ad512120b4f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:39.401037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.401004 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.401258 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.401242 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8gz\" (UniqueName: \"kubernetes.io/projected/c239f9a9-c47e-440c-874f-ad512120b4f2-kube-api-access-2r8gz\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.401353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.401342 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c239f9a9-c47e-440c-874f-ad512120b4f2-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.401441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.401431 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239f9a9-c47e-440c-874f-ad512120b4f2-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.656478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:39.656438 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:45:40.128012 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:40.127979 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" Oct 29 22:45:40.128497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:40.128022 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g" event={"ID":"c239f9a9-c47e-440c-874f-ad512120b4f2","Type":"ContainerDied","Data":"01fd1c006127036aab281138cae74bf90cf7f6bc006efc58664f77107b0c3600"} Oct 29 22:45:40.128497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:40.128084 4376 scope.go:122] "RemoveContainer" containerID="c96f6a1e529b1a366a876cab64d13247962dd3076743edb027a318470352194d" Oct 29 22:45:40.195988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:40.195881 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:45:40.203243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:40.201150 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4afaba-7ccddc69b9-bfr5g"] Oct 29 22:45:41.272440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:41.272402 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c239f9a9-c47e-440c-874f-ad512120b4f2" path="/var/lib/kubelet/pods/c239f9a9-c47e-440c-874f-ad512120b4f2/volumes" Oct 29 22:45:43.105131 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:43.104825 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:43.107028 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:43.106221 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:46.435275 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:46.435239 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:45:46.803512 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:46.803473 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:45:46.803773 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:46.803743 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" containerID="cri-o://413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404" gracePeriod=30 Oct 29 22:45:48.597209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:48.597168 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:45:48.597656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:48.597483 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" containerID="cri-o://6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f" gracePeriod=30 Oct 29 22:45:51.317732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:51.317691 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:51.318246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:51.317747 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:52.075417 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.075356 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:45:52.191741 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.191706 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.191759 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.191893 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192004 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192083 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192185 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192191 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192224 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192275 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192340 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\") pod \"51391134-c2b5-4bdf-b54f-971a421cfcaf\" (UID: \"51391134-c2b5-4bdf-b54f-971a421cfcaf\") " Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192220 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192403 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192474 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192600 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192896 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192955 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192969 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192981 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51391134-c2b5-4bdf-b54f-971a421cfcaf-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.194685 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.192994 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.195817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.194774 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:52.195817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.194963 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "kube-api-access-75kg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:52.195817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.195129 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:52.195817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.195249 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy" pod "51391134-c2b5-4bdf-b54f-971a421cfcaf" (UID: "51391134-c2b5-4bdf-b54f-971a421cfcaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.212499 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.212467 4376 generic.go:356] "Generic (PLEG): container finished" podID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerID="413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404" exitCode=0 Oct 29 22:45:52.212633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.212503 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerDied","Data":"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404"} Oct 29 22:45:52.212633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.212533 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" Oct 29 22:45:52.212633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.212543 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8" event={"ID":"51391134-c2b5-4bdf-b54f-971a421cfcaf","Type":"ContainerDied","Data":"11e7d355184efdd7bf81c08f6b1adbca8389e785ddcd4d557cd5db3c5b17c123"} Oct 29 22:45:52.212633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.212566 4376 scope.go:122] "RemoveContainer" containerID="413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404" Oct 29 22:45:52.223422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.223404 4376 scope.go:122] "RemoveContainer" containerID="049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88" Oct 29 22:45:52.232519 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.232500 4376 scope.go:122] "RemoveContainer" containerID="413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404" Oct 29 22:45:52.232757 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:52.232740 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404\": container with ID starting with 413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404 not found: ID does not exist" containerID="413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404" Oct 29 22:45:52.232803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.232763 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404"} err="failed to get container status \"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404\": rpc error: code = NotFound desc = could not find container \"413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404\": container with ID starting with 413964abec3b455097a78ee01d24c09d1aa9619c862a81be42b6348c68177404 not found: ID does not exist" Oct 29 22:45:52.232803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.232780 4376 scope.go:122] "RemoveContainer" containerID="049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88" Oct 29 22:45:52.233005 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:52.232988 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88\": container with ID starting with 049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88 not found: ID does not exist" containerID="049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88" Oct 29 22:45:52.233052 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.233010 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88"} err="failed to get container status \"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88\": rpc error: code = NotFound desc = could not find container \"049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88\": container with ID starting with 049cb1c09a666b9cd7543bd3fb1a212f8fd54ed73a401b7ed7ffffb4ec142d88 not found: ID does not exist" Oct 29 22:45:52.268807 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.268782 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:45:52.274122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.274100 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-cjfl8"] Oct 29 22:45:52.294344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.294320 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kg5\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-kube-api-access-75kg5\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.294344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.294343 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.294493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.294362 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.294493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:52.294371 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51391134-c2b5-4bdf-b54f-971a421cfcaf-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.104057 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.104001 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:53.104504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.104029 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:53.272033 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.271999 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" path="/var/lib/kubelet/pods/51391134-c2b5-4bdf-b54f-971a421cfcaf/volumes" Oct 29 22:45:53.875551 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.875488 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:45:53.913296 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913267 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913307 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913336 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913378 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913396 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913430 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913491 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913595 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913620 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:53.913881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913726 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\") pod \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\" (UID: \"e58c4821-b7f6-4b5b-8215-48c20db90b6b\") " Oct 29 22:45:53.913881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913744 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:53.913881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913774 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:53.913881 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.913786 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:53.914120 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914030 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:53.914452 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914433 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.914541 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914455 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.914541 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914473 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.914541 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914487 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.914541 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.914500 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:53.915872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.915840 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:53.915991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.915894 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "kube-api-access-xt42b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:53.916070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.916036 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:53.916123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:53.916060 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy" pod "e58c4821-b7f6-4b5b-8215-48c20db90b6b" (UID: "e58c4821-b7f6-4b5b-8215-48c20db90b6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:54.015495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.015463 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:54.015495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.015494 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt42b\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-kube-api-access-xt42b\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:54.015495 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.015504 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:54.015728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.015520 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58c4821-b7f6-4b5b-8215-48c20db90b6b-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:54.229044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.229007 4376 generic.go:356] "Generic (PLEG): container finished" podID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerID="6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f" exitCode=0 Oct 29 22:45:54.229503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.229061 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" Oct 29 22:45:54.229503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.229092 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerDied","Data":"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f"} Oct 29 22:45:54.229503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.229150 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf" event={"ID":"e58c4821-b7f6-4b5b-8215-48c20db90b6b","Type":"ContainerDied","Data":"4e392ed58d33ae04039e430c6dbe1011a25d192939a29886eea6faab8076bc9f"} Oct 29 22:45:54.229503 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.229184 4376 scope.go:122] "RemoveContainer" containerID="6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f" Oct 29 22:45:54.240983 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.240964 4376 scope.go:122] "RemoveContainer" containerID="247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c" Oct 29 22:45:54.250374 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.250354 4376 scope.go:122] "RemoveContainer" containerID="6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f" Oct 29 22:45:54.250620 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:54.250601 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f\": container with ID starting with 6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f not found: ID does not exist" containerID="6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f" Oct 29 22:45:54.250683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.250627 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f"} err="failed to get container status \"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f\": rpc error: code = NotFound desc = could not find container \"6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f\": container with ID starting with 6037a06058d2685f287bb62f3faab6d4b932078d3bbf30d73125ce621ec3723f not found: ID does not exist" Oct 29 22:45:54.250683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.250658 4376 scope.go:122] "RemoveContainer" containerID="247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c" Oct 29 22:45:54.250912 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:54.250884 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c\": container with ID starting with 247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c not found: ID does not exist" containerID="247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c" Oct 29 22:45:54.250965 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.250918 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c"} err="failed to get container status \"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c\": rpc error: code = NotFound desc = could not find container \"247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c\": container with ID starting with 247572e67c249a22b12e12332e61a8e54004dafa0c0cef52817ce107f4361a9c not found: ID does not exist" Oct 29 22:45:54.286676 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.286642 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:45:54.299456 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.299425 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4afaba-openshift-default-7cc7f787c7-dkcdf"] Oct 29 22:45:54.302989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.302966 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:45:54.679204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.679124 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" podUID="ec83798f-31a3-4d39-8de9-16f11610491f" containerName="main" containerID="cri-o://e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e" gracePeriod=45 Oct 29 22:45:54.988321 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:54.988288 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:45:55.031697 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.031668 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.031850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.031766 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.031850 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.031838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.032000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.031970 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.032000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.031981 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\" (UniqueName: \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.032111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032018 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.032111 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032039 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\") pod \"ec83798f-31a3-4d39-8de9-16f11610491f\" (UID: \"ec83798f-31a3-4d39-8de9-16f11610491f\") " Oct 29 22:45:55.032240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032139 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.032240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032157 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.032633 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032610 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.032880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032857 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.032880 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.032882 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.034879 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.034854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:55.035136 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.035098 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "kube-api-access-m9dzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.041473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.041449 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm" pod "ec83798f-31a3-4d39-8de9-16f11610491f" (UID: "ec83798f-31a3-4d39-8de9-16f11610491f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:55.134614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.134584 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec83798f-31a3-4d39-8de9-16f11610491f-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.134614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.134610 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dzb\" (UniqueName: \"kubernetes.io/projected/ec83798f-31a3-4d39-8de9-16f11610491f-kube-api-access-m9dzb\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.134614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.134620 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec83798f-31a3-4d39-8de9-16f11610491f-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.238257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.238174 4376 generic.go:356] "Generic (PLEG): container finished" podID="ec83798f-31a3-4d39-8de9-16f11610491f" containerID="e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e" exitCode=0 Oct 29 22:45:55.238257 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.238221 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" Oct 29 22:45:55.238723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.238260 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" event={"ID":"ec83798f-31a3-4d39-8de9-16f11610491f","Type":"ContainerDied","Data":"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e"} Oct 29 22:45:55.238723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.238298 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5" event={"ID":"ec83798f-31a3-4d39-8de9-16f11610491f","Type":"ContainerDied","Data":"e6f3e157e6b42a4d06ae930b833c32a665c1f32193c3215775a07d9594bb1fd0"} Oct 29 22:45:55.238723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.238317 4376 scope.go:122] "RemoveContainer" containerID="e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e" Oct 29 22:45:55.248568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.248550 4376 scope.go:122] "RemoveContainer" containerID="e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e" Oct 29 22:45:55.248847 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:55.248828 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e\": container with ID starting with e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e not found: ID does not exist" containerID="e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e" Oct 29 22:45:55.248894 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.248854 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e"} err="failed to get container status \"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e\": rpc error: code = NotFound desc = could not find container \"e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e\": container with ID starting with e480be7ff6e91eabe4d4c9833899a9d14197c46bcfd194de499dd49b585c6f2e not found: ID does not exist" Oct 29 22:45:55.271688 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.271662 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" path="/var/lib/kubelet/pods/e58c4821-b7f6-4b5b-8215-48c20db90b6b/volumes" Oct 29 22:45:55.293813 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.293790 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:45:55.298334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.298315 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-27b6db/test-model-27b6db-kserve-f8d98977c-p9sl5"] Oct 29 22:45:55.629154 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.629120 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:45:55.676760 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.676729 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:45:55.742162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742129 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\" (UniqueName: \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\") pod \"c3443574-dc80-4ca9-b8c1-8c052afaf411\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " Oct 29 22:45:55.742324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\" (UniqueName: \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\") pod \"be560f2c-f950-48ba-97d8-4f2a112d590c\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " Oct 29 22:45:55.742324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742229 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\") pod \"c3443574-dc80-4ca9-b8c1-8c052afaf411\" (UID: \"c3443574-dc80-4ca9-b8c1-8c052afaf411\") " Oct 29 22:45:55.742324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742282 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\") pod \"be560f2c-f950-48ba-97d8-4f2a112d590c\" (UID: \"be560f2c-f950-48ba-97d8-4f2a112d590c\") " Oct 29 22:45:55.742710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742679 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume" pod "c3443574-dc80-4ca9-b8c1-8c052afaf411" (UID: "c3443574-dc80-4ca9-b8c1-8c052afaf411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:55.742710 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742679 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume" pod "be560f2c-f950-48ba-97d8-4f2a112d590c" (UID: "be560f2c-f950-48ba-97d8-4f2a112d590c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:55.742969 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742945 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3443574-dc80-4ca9-b8c1-8c052afaf411-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.743044 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.742975 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be560f2c-f950-48ba-97d8-4f2a112d590c-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.744529 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.744496 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr" pod "c3443574-dc80-4ca9-b8c1-8c052afaf411" (UID: "c3443574-dc80-4ca9-b8c1-8c052afaf411"). InnerVolumeSpecName "kube-api-access-th2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.744687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.744664 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4" pod "be560f2c-f950-48ba-97d8-4f2a112d590c" (UID: "be560f2c-f950-48ba-97d8-4f2a112d590c"). InnerVolumeSpecName "kube-api-access-mkdm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:55.844372 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.844342 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2mr\" (UniqueName: \"kubernetes.io/projected/c3443574-dc80-4ca9-b8c1-8c052afaf411-kube-api-access-th2mr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:55.844527 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:55.844370 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdm4\" (UniqueName: \"kubernetes.io/projected/be560f2c-f950-48ba-97d8-4f2a112d590c-kube-api-access-mkdm4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:56.245238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.245200 4376 generic.go:356] "Generic (PLEG): container finished" podID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerID="e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad" exitCode=0 Oct 29 22:45:56.245238 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.245245 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" Oct 29 22:45:56.245799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.245279 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" event={"ID":"be560f2c-f950-48ba-97d8-4f2a112d590c","Type":"ContainerDied","Data":"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad"} Oct 29 22:45:56.245799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.245381 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7" event={"ID":"be560f2c-f950-48ba-97d8-4f2a112d590c","Type":"ContainerDied","Data":"f9edb909f332744236c1fb3f7dd0eb9b5c881b89093e2b99bc78580bccb568ae"} Oct 29 22:45:56.245799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.245417 4376 scope.go:122] "RemoveContainer" containerID="e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad" Oct 29 22:45:56.248487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.248461 4376 generic.go:356] "Generic (PLEG): container finished" podID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerID="0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4" exitCode=0 Oct 29 22:45:56.248630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.248504 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" Oct 29 22:45:56.248630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.248528 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" event={"ID":"c3443574-dc80-4ca9-b8c1-8c052afaf411","Type":"ContainerDied","Data":"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4"} Oct 29 22:45:56.248630 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.248582 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj" event={"ID":"c3443574-dc80-4ca9-b8c1-8c052afaf411","Type":"ContainerDied","Data":"2352af2fbc0be2fa8d78371ee91fef623a223f6ab416f92b409b638c415b7fd2"} Oct 29 22:45:56.258045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.258025 4376 scope.go:122] "RemoveContainer" containerID="e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad" Oct 29 22:45:56.258372 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:56.258333 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad\": container with ID starting with e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad not found: ID does not exist" containerID="e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad" Oct 29 22:45:56.258463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.258381 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad"} err="failed to get container status \"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad\": rpc error: code = NotFound desc = could not find container \"e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad\": container with ID starting with e560072e9887ac8537f92522f3bacc45cb8deb05e6366504949b14ca656d2cad not found: ID does not exist" Oct 29 22:45:56.258463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.258406 4376 scope.go:122] "RemoveContainer" containerID="0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4" Oct 29 22:45:56.267440 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.267424 4376 scope.go:122] "RemoveContainer" containerID="0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4" Oct 29 22:45:56.267690 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:45:56.267670 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4\": container with ID starting with 0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4 not found: ID does not exist" containerID="0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4" Oct 29 22:45:56.267757 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.267695 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4"} err="failed to get container status \"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4\": rpc error: code = NotFound desc = could not find container \"0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4\": container with ID starting with 0a5907f5b5f00a1af39da1f1d97c0ef98db9ebd42536e8f3b6421df3084083a4 not found: ID does not exist" Oct 29 22:45:56.300769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.300746 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:45:56.313254 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.313226 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d09411-644878db47-xz2m7"] Oct 29 22:45:56.342997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.342971 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:45:56.349769 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:56.349742 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d09411-6f79df74f4-lz6qj"] Oct 29 22:45:57.271827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:57.271794 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" path="/var/lib/kubelet/pods/be560f2c-f950-48ba-97d8-4f2a112d590c/volumes" Oct 29 22:45:57.272267 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:57.272251 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" path="/var/lib/kubelet/pods/c3443574-dc80-4ca9-b8c1-8c052afaf411/volumes" Oct 29 22:45:57.272664 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:57.272651 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec83798f-31a3-4d39-8de9-16f11610491f" path="/var/lib/kubelet/pods/ec83798f-31a3-4d39-8de9-16f11610491f/volumes" Oct 29 22:45:57.944421 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:45:57.944380 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:46:02.945210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:02.945164 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" containerID="cri-o://055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed" gracePeriod=25 Oct 29 22:46:05.126504 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:05.126427 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:46:07.463637 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.463600 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:46:07.464368 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.464321 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerName="maas-api" containerID="cri-o://90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f" gracePeriod=30 Oct 29 22:46:07.769596 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.769571 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:46:07.922758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.922724 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\") pod \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " Oct 29 22:46:07.922966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.922827 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\" (UniqueName: \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\") pod \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " Oct 29 22:46:07.922966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.922875 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\") pod \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " Oct 29 22:46:07.923069 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.922974 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\") pod \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\" (UID: \"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4\") " Oct 29 22:46:07.923298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.923264 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca" pod "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" (UID: "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:07.923507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.923492 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:07.925084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.925063 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd" pod "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" (UID: "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4"). InnerVolumeSpecName "kube-api-access-9pmvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:07.925347 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.925331 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs" pod "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" (UID: "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:07.925411 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:07.925353 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls" pod "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" (UID: "9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:08.024509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.024417 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:08.024509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.024453 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmvd\" (UniqueName: \"kubernetes.io/projected/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-kube-api-access-9pmvd\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:08.024509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.024467 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:08.327511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.327429 4376 generic.go:356] "Generic (PLEG): container finished" podID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerID="90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f" exitCode=0 Oct 29 22:46:08.327511 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.327480 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" Oct 29 22:46:08.327733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.327511 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" event={"ID":"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4","Type":"ContainerDied","Data":"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f"} Oct 29 22:46:08.327733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.327566 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" event={"ID":"9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4","Type":"ContainerDied","Data":"97e2a985008509d84428a24bfa95b6b32f65b3bd72f7579257ac7f1458607476"} Oct 29 22:46:08.327733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.327589 4376 scope.go:122] "RemoveContainer" containerID="90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f" Oct 29 22:46:08.338339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.338322 4376 scope.go:122] "RemoveContainer" containerID="90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f" Oct 29 22:46:08.338613 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:08.338583 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f\": container with ID starting with 90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f not found: ID does not exist" containerID="90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f" Oct 29 22:46:08.338659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.338622 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f"} err="failed to get container status \"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f\": rpc error: code = NotFound desc = could not find container \"90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f\": container with ID starting with 90bfa53abb3ec8bb2cc2860b14fea83e418e687240adf9fb0e08812af16d6e4f not found: ID does not exist" Oct 29 22:46:08.385199 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.385176 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:46:08.397113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:08.397092 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr"] Oct 29 22:46:09.271487 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.271452 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" path="/var/lib/kubelet/pods/9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4/volumes" Oct 29 22:46:09.323827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.323789 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" podUID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" containerName="main" containerID="cri-o://a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31" gracePeriod=45 Oct 29 22:46:09.590621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.590594 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:46:09.638379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638415 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638566 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638481 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\" (UniqueName: \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638571 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638617 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:09.638715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638689 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\") pod \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\" (UID: \"afa00f6c-e1bd-41b7-96fe-cacd076832d8\") " Oct 29 22:46:09.638887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.638821 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:09.639299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.639265 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:09.639299 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.639294 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:09.639463 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.639422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:09.641140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.641111 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "kube-api-access-r5km4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:09.641270 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.641165 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:09.641643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.641616 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs" pod "afa00f6c-e1bd-41b7-96fe-cacd076832d8" (UID: "afa00f6c-e1bd-41b7-96fe-cacd076832d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:09.740624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.740595 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5km4\" (UniqueName: \"kubernetes.io/projected/afa00f6c-e1bd-41b7-96fe-cacd076832d8-kube-api-access-r5km4\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:09.740624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.740619 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:09.740787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.740631 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa00f6c-e1bd-41b7-96fe-cacd076832d8-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:09.740787 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:09.740640 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa00f6c-e1bd-41b7-96fe-cacd076832d8-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:10.063284 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.063240 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:10.127063 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.127020 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" containerID="cri-o://11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e" gracePeriod=25 Oct 29 22:46:10.352953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.352830 4376 generic.go:356] "Generic (PLEG): container finished" podID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" containerID="a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31" exitCode=0 Oct 29 22:46:10.352953 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.352886 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" Oct 29 22:46:10.353507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.352891 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" event={"ID":"afa00f6c-e1bd-41b7-96fe-cacd076832d8","Type":"ContainerDied","Data":"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31"} Oct 29 22:46:10.353507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.353026 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74" event={"ID":"afa00f6c-e1bd-41b7-96fe-cacd076832d8","Type":"ContainerDied","Data":"4c6fab10a5e53a81792445e0e8b0238c9871191f5e872b2414e00ee24a3adbc1"} Oct 29 22:46:10.353507 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.353043 4376 scope.go:122] "RemoveContainer" containerID="a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31" Oct 29 22:46:10.363481 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.363463 4376 scope.go:122] "RemoveContainer" containerID="a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31" Oct 29 22:46:10.363729 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:10.363711 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31\": container with ID starting with a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31 not found: ID does not exist" containerID="a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31" Oct 29 22:46:10.363770 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.363736 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31"} err="failed to get container status \"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31\": rpc error: code = NotFound desc = could not find container \"a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31\": container with ID starting with a671fcd28983952421edd5ad6973bb2cabdd45d7e709a301fe40babab4df2a31 not found: ID does not exist" Oct 29 22:46:10.410255 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.410228 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:46:10.423235 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.423207 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-580af2/test-model-580af2-kserve-768c9f789f-n8n74"] Oct 29 22:46:10.641333 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:10.641238 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f52917-75cc4fbb5f-2h4lr" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.172:8443/health\": context deadline exceeded" Oct 29 22:46:11.171396 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.171356 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:46:11.171838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.171808 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" podUID="533d11b2-d4bb-48eb-b462-81732dee78e9" containerName="maas-api" containerID="cri-o://aa794dec46dea6b9b2a105daf5d76b93ccc08e621e0a3863303d71e1e0c2ab4b" gracePeriod=30 Oct 29 22:46:11.273119 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.273085 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" path="/var/lib/kubelet/pods/afa00f6c-e1bd-41b7-96fe-cacd076832d8/volumes" Oct 29 22:46:11.361716 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.361682 4376 generic.go:356] "Generic (PLEG): container finished" podID="533d11b2-d4bb-48eb-b462-81732dee78e9" containerID="aa794dec46dea6b9b2a105daf5d76b93ccc08e621e0a3863303d71e1e0c2ab4b" exitCode=0 Oct 29 22:46:11.362163 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.361754 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" event={"ID":"533d11b2-d4bb-48eb-b462-81732dee78e9","Type":"ContainerDied","Data":"aa794dec46dea6b9b2a105daf5d76b93ccc08e621e0a3863303d71e1e0c2ab4b"} Oct 29 22:46:11.459848 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.459817 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:46:11.561925 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.561870 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\") pod \"533d11b2-d4bb-48eb-b462-81732dee78e9\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " Oct 29 22:46:11.562898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.562868 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\") pod \"533d11b2-d4bb-48eb-b462-81732dee78e9\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " Oct 29 22:46:11.563375 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.563356 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\") pod \"533d11b2-d4bb-48eb-b462-81732dee78e9\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " Oct 29 22:46:11.563549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.563530 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\" (UniqueName: \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\") pod \"533d11b2-d4bb-48eb-b462-81732dee78e9\" (UID: \"533d11b2-d4bb-48eb-b462-81732dee78e9\") " Oct 29 22:46:11.564129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.563820 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca" pod "533d11b2-d4bb-48eb-b462-81732dee78e9" (UID: "533d11b2-d4bb-48eb-b462-81732dee78e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:11.571310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.571271 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533d11b2-d4bb-48eb-b462-81732dee78e9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.579151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.577440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs" pod "533d11b2-d4bb-48eb-b462-81732dee78e9" (UID: "533d11b2-d4bb-48eb-b462-81732dee78e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:11.579151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.577738 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w" pod "533d11b2-d4bb-48eb-b462-81732dee78e9" (UID: "533d11b2-d4bb-48eb-b462-81732dee78e9"). InnerVolumeSpecName "kube-api-access-ndj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:11.579151 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.577886 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls" pod "533d11b2-d4bb-48eb-b462-81732dee78e9" (UID: "533d11b2-d4bb-48eb-b462-81732dee78e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:11.673241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.673211 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndj9w\" (UniqueName: \"kubernetes.io/projected/533d11b2-d4bb-48eb-b462-81732dee78e9-kube-api-access-ndj9w\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.673241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.673239 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:11.673441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:11.673249 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533d11b2-d4bb-48eb-b462-81732dee78e9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:12.370231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:12.370201 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" Oct 29 22:46:12.370231 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:12.370212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b" event={"ID":"533d11b2-d4bb-48eb-b462-81732dee78e9","Type":"ContainerDied","Data":"f79420380ebcc080a79e1f34c854215328a58ad1b7f7b7890a823888cd11bfac"} Oct 29 22:46:12.370762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:12.370276 4376 scope.go:122] "RemoveContainer" containerID="aa794dec46dea6b9b2a105daf5d76b93ccc08e621e0a3863303d71e1e0c2ab4b" Oct 29 22:46:12.426381 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:12.426346 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:46:12.438414 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:12.438384 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-563f39-7d55c55667-hzl5b"] Oct 29 22:46:13.272966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:13.272930 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533d11b2-d4bb-48eb-b462-81732dee78e9" path="/var/lib/kubelet/pods/533d11b2-d4bb-48eb-b462-81732dee78e9/volumes" Oct 29 22:46:13.549433 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:13.549342 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:16.434965 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:16.434891 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:46:20.062640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:20.062593 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:21.949336 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:21.949302 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 22:46:21.962930 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:21.962880 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 22:46:23.209530 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.209505 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:46:23.357623 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.357583 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\") pod \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " Oct 29 22:46:23.357822 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.357806 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\" (UniqueName: \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\") pod \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\" (UID: \"04b0d153-a9a9-40fb-9627-02d86d2b68a2\") " Oct 29 22:46:23.358067 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.358033 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume" pod "04b0d153-a9a9-40fb-9627-02d86d2b68a2" (UID: "04b0d153-a9a9-40fb-9627-02d86d2b68a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:23.358367 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.358326 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b0d153-a9a9-40fb-9627-02d86d2b68a2-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:23.360030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.360003 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg" pod "04b0d153-a9a9-40fb-9627-02d86d2b68a2" (UID: "04b0d153-a9a9-40fb-9627-02d86d2b68a2"). InnerVolumeSpecName "kube-api-access-z4dbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:23.443847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.443809 4376 generic.go:356] "Generic (PLEG): container finished" podID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerID="055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed" exitCode=0 Oct 29 22:46:23.444086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.443857 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" Oct 29 22:46:23.444086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.443898 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" event={"ID":"04b0d153-a9a9-40fb-9627-02d86d2b68a2","Type":"ContainerDied","Data":"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed"} Oct 29 22:46:23.444086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.443984 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s" event={"ID":"04b0d153-a9a9-40fb-9627-02d86d2b68a2","Type":"ContainerDied","Data":"e4f10d79e12cfbcd692df49eab3c91af327e47e81446929b3fff50fada268a53"} Oct 29 22:46:23.444086 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.444018 4376 scope.go:122] "RemoveContainer" containerID="055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed" Oct 29 22:46:23.455204 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.455179 4376 scope.go:122] "RemoveContainer" containerID="055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed" Oct 29 22:46:23.455564 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:23.455543 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed\": container with ID starting with 055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed not found: ID does not exist" containerID="055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed" Oct 29 22:46:23.455614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.455574 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed"} err="failed to get container status \"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed\": rpc error: code = NotFound desc = could not find container \"055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed\": container with ID starting with 055fb4b94801fd2e096ecb822858a7d76fc61380057754295e91678d862bf7ed not found: ID does not exist" Oct 29 22:46:23.459646 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.459587 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dbg\" (UniqueName: \"kubernetes.io/projected/04b0d153-a9a9-40fb-9627-02d86d2b68a2-kube-api-access-z4dbg\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:23.499383 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.499346 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:46:23.504934 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.504870 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-563f39-78d468947f-77s9s"] Oct 29 22:46:23.548771 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:23.548734 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:25.272478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:25.272440 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" path="/var/lib/kubelet/pods/04b0d153-a9a9-40fb-9627-02d86d2b68a2/volumes" Oct 29 22:46:30.395824 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.394698 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:46:30.448744 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.448714 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\" (UniqueName: \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\") pod \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " Oct 29 22:46:30.448928 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.448854 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\") pod \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\" (UID: \"439a7a25-91d7-4d82-a5bd-93cb94f10eed\") " Oct 29 22:46:30.449314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.449276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume" pod "439a7a25-91d7-4d82-a5bd-93cb94f10eed" (UID: "439a7a25-91d7-4d82-a5bd-93cb94f10eed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:30.450924 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.450885 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf" pod "439a7a25-91d7-4d82-a5bd-93cb94f10eed" (UID: "439a7a25-91d7-4d82-a5bd-93cb94f10eed"). InnerVolumeSpecName "kube-api-access-m8rjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:30.507128 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.507090 4376 generic.go:356] "Generic (PLEG): container finished" podID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerID="11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e" exitCode=0 Oct 29 22:46:30.507320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.507138 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" Oct 29 22:46:30.507320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.507177 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" event={"ID":"439a7a25-91d7-4d82-a5bd-93cb94f10eed","Type":"ContainerDied","Data":"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e"} Oct 29 22:46:30.507320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.507246 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst" event={"ID":"439a7a25-91d7-4d82-a5bd-93cb94f10eed","Type":"ContainerDied","Data":"ced9376f073f748ac3d2c3190bbbeb5ce834c1138aacaebc2b791d28f04fde3d"} Oct 29 22:46:30.507320 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.507280 4376 scope.go:122] "RemoveContainer" containerID="11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e" Oct 29 22:46:30.517584 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.517565 4376 scope.go:122] "RemoveContainer" containerID="11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e" Oct 29 22:46:30.517844 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:30.517823 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e\": container with ID starting with 11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e not found: ID does not exist" containerID="11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e" Oct 29 22:46:30.517892 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.517853 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e"} err="failed to get container status \"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e\": rpc error: code = NotFound desc = could not find container \"11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e\": container with ID starting with 11b922bab323f40dff07602165dcda939355c7b6cf0f33eebcbc1c35109ffb4e not found: ID does not exist" Oct 29 22:46:30.549981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.549959 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rjf\" (UniqueName: \"kubernetes.io/projected/439a7a25-91d7-4d82-a5bd-93cb94f10eed-kube-api-access-m8rjf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.549981 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.549980 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a7a25-91d7-4d82-a5bd-93cb94f10eed-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.562364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.562330 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:46:30.567602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:30.567580 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27b6db-798fd8df9-mwpst"] Oct 29 22:46:31.272211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:31.272176 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" path="/var/lib/kubelet/pods/439a7a25-91d7-4d82-a5bd-93cb94f10eed/volumes" Oct 29 22:46:34.232895 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.232854 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:46:34.233513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.233480 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" podUID="a401e66c-872c-4d84-bf69-213932b8cd09" containerName="maas-api" containerID="cri-o://1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809" gracePeriod=30 Oct 29 22:46:34.531425 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.531395 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:46:34.548109 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.548081 4376 generic.go:356] "Generic (PLEG): container finished" podID="a401e66c-872c-4d84-bf69-213932b8cd09" containerID="1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809" exitCode=0 Oct 29 22:46:34.548271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.548126 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" Oct 29 22:46:34.548271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.548164 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" event={"ID":"a401e66c-872c-4d84-bf69-213932b8cd09","Type":"ContainerDied","Data":"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809"} Oct 29 22:46:34.548271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.548222 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z" event={"ID":"a401e66c-872c-4d84-bf69-213932b8cd09","Type":"ContainerDied","Data":"a4b6e716a18d47eb956e178358717a424062b678975a69cb99c8f2d95c8b1be4"} Oct 29 22:46:34.548271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.548245 4376 scope.go:122] "RemoveContainer" containerID="1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809" Oct 29 22:46:34.557438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.557407 4376 scope.go:122] "RemoveContainer" containerID="1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809" Oct 29 22:46:34.557732 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:34.557709 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809\": container with ID starting with 1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809 not found: ID does not exist" containerID="1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809" Oct 29 22:46:34.557800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.557745 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809"} err="failed to get container status \"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809\": rpc error: code = NotFound desc = could not find container \"1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809\": container with ID starting with 1244af1682d5d79bed1e17eae223a44bec88926c24dc0a27f404680f7eabb809 not found: ID does not exist" Oct 29 22:46:34.605555 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.605515 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\" (UniqueName: \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\") pod \"a401e66c-872c-4d84-bf69-213932b8cd09\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " Oct 29 22:46:34.605829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.605616 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\") pod \"a401e66c-872c-4d84-bf69-213932b8cd09\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " Oct 29 22:46:34.605829 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.605745 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\") pod \"a401e66c-872c-4d84-bf69-213932b8cd09\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " Oct 29 22:46:34.606303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.606263 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\") pod \"a401e66c-872c-4d84-bf69-213932b8cd09\" (UID: \"a401e66c-872c-4d84-bf69-213932b8cd09\") " Oct 29 22:46:34.606670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.606647 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca" pod "a401e66c-872c-4d84-bf69-213932b8cd09" (UID: "a401e66c-872c-4d84-bf69-213932b8cd09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:34.606962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.606942 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a401e66c-872c-4d84-bf69-213932b8cd09-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:34.613085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.613056 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls" pod "a401e66c-872c-4d84-bf69-213932b8cd09" (UID: "a401e66c-872c-4d84-bf69-213932b8cd09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:34.613228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.613176 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs" pod "a401e66c-872c-4d84-bf69-213932b8cd09" (UID: "a401e66c-872c-4d84-bf69-213932b8cd09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:34.628478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.628427 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9" pod "a401e66c-872c-4d84-bf69-213932b8cd09" (UID: "a401e66c-872c-4d84-bf69-213932b8cd09"). InnerVolumeSpecName "kube-api-access-2rns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:34.708510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.708473 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rns9\" (UniqueName: \"kubernetes.io/projected/a401e66c-872c-4d84-bf69-213932b8cd09-kube-api-access-2rns9\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:34.708510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.708503 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:34.708510 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.708512 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a401e66c-872c-4d84-bf69-213932b8cd09-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:34.906438 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.906405 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:46:34.918628 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:34.918596 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c3d9a-77f4dc574b-tvs4z"] Oct 29 22:46:35.271977 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:35.271934 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a401e66c-872c-4d84-bf69-213932b8cd09" path="/var/lib/kubelet/pods/a401e66c-872c-4d84-bf69-213932b8cd09/volumes" Oct 29 22:46:36.480814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.479468 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:46:36.480814 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.479731 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" podUID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" containerName="maas-api" containerID="cri-o://83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a" gracePeriod=30 Oct 29 22:46:36.753831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.753803 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:46:36.839420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.839379 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\" (UniqueName: \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\") pod \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " Oct 29 22:46:36.839643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.839473 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\") pod \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " Oct 29 22:46:36.839643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.839509 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\") pod \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " Oct 29 22:46:36.839794 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.839686 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\") pod \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\" (UID: \"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87\") " Oct 29 22:46:36.840138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.840105 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca" pod "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" (UID: "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:36.840281 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.840232 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:36.842005 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.841977 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq" pod "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" (UID: "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87"). InnerVolumeSpecName "kube-api-access-79gdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:36.842113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.842004 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls" pod "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" (UID: "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:36.842155 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.842132 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs" pod "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" (UID: "9a2fba5c-c2b7-44d1-87fd-12d623f8cc87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:36.942138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.942096 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gdq\" (UniqueName: \"kubernetes.io/projected/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-kube-api-access-79gdq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:36.942138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.942137 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:36.942454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.942152 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:36.978659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.978588 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" containerID="cri-o://bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b" gracePeriod=45 Oct 29 22:46:36.985344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:36.985231 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" containerID="cri-o://87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a" gracePeriod=45 Oct 29 22:46:37.085799 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.085758 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": dial tcp 10.134.0.75:8000: connect: connection refused" Oct 29 22:46:37.269308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.269277 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:46:37.303366 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.303334 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:46:37.356268 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356231 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356282 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356388 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\" (UniqueName: \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356479 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356486 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356516 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\") pod \"f2a90d29-40a0-4aa8-9991-61f234b3e273\" (UID: \"f2a90d29-40a0-4aa8-9991-61f234b3e273\") " Oct 29 22:46:37.356683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.356875 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356803 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.356988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.356965 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.357315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.357279 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.357315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.357306 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.359034 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.358938 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.359173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.359147 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.359173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.359151 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq" pod "f2a90d29-40a0-4aa8-9991-61f234b3e273" (UID: "f2a90d29-40a0-4aa8-9991-61f234b3e273"). InnerVolumeSpecName "kube-api-access-zzvkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.458477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458437 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.458656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458518 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\" (UniqueName: \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.458656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458558 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.458656 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458590 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.458949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458697 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.458949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458739 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.458949 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.458757 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\") pod \"ee972f81-b171-40b5-8642-a8f34172e90d\" (UID: \"ee972f81-b171-40b5-8642-a8f34172e90d\") " Oct 29 22:46:37.459138 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459000 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.459216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.459419 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459396 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459424 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459443 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzvkq\" (UniqueName: \"kubernetes.io/projected/f2a90d29-40a0-4aa8-9991-61f234b3e273-kube-api-access-zzvkq\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459457 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459470 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459482 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a90d29-40a0-4aa8-9991-61f234b3e273-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.459715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.459537 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a90d29-40a0-4aa8-9991-61f234b3e273-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.461050 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.461024 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.461282 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.461260 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "kube-api-access-7j8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.461661 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.461641 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs" pod "ee972f81-b171-40b5-8642-a8f34172e90d" (UID: "ee972f81-b171-40b5-8642-a8f34172e90d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.561123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.561046 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee972f81-b171-40b5-8642-a8f34172e90d-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.561123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.561076 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee972f81-b171-40b5-8642-a8f34172e90d-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.561123 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.561085 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8sx\" (UniqueName: \"kubernetes.io/projected/ee972f81-b171-40b5-8642-a8f34172e90d-kube-api-access-7j8sx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.566379 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.566344 4376 generic.go:356] "Generic (PLEG): container finished" podID="ee972f81-b171-40b5-8642-a8f34172e90d" containerID="87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a" exitCode=0 Oct 29 22:46:37.566525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.566397 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" Oct 29 22:46:37.566525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.566431 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" event={"ID":"ee972f81-b171-40b5-8642-a8f34172e90d","Type":"ContainerDied","Data":"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a"} Oct 29 22:46:37.566525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.566474 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" event={"ID":"ee972f81-b171-40b5-8642-a8f34172e90d","Type":"ContainerDied","Data":"b6decea1c7964661de51ab0b4e12686c86ce9ee2d7e21d47eba87f93f9e62593"} Oct 29 22:46:37.566525 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.566495 4376 scope.go:122] "RemoveContainer" containerID="87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a" Oct 29 22:46:37.568004 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.567982 4376 generic.go:356] "Generic (PLEG): container finished" podID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" containerID="83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a" exitCode=0 Oct 29 22:46:37.568118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.568019 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" event={"ID":"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87","Type":"ContainerDied","Data":"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a"} Oct 29 22:46:37.568118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.568079 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" event={"ID":"9a2fba5c-c2b7-44d1-87fd-12d623f8cc87","Type":"ContainerDied","Data":"b1ddbd9bb3aaf64b31a91660aea32a9a8b95c0d72dbb77f4551de815e04fe3d8"} Oct 29 22:46:37.568118 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.568034 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm" Oct 29 22:46:37.569554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.569532 4376 generic.go:356] "Generic (PLEG): container finished" podID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerID="bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b" exitCode=0 Oct 29 22:46:37.569672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.569568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" event={"ID":"f2a90d29-40a0-4aa8-9991-61f234b3e273","Type":"ContainerDied","Data":"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b"} Oct 29 22:46:37.569672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.569608 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" event={"ID":"f2a90d29-40a0-4aa8-9991-61f234b3e273","Type":"ContainerDied","Data":"4643ef0ce7e572c923f6e0f7c2911e3d9c6b771fab23cf6b3d9d12493487148a"} Oct 29 22:46:37.569672 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.569576 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r" Oct 29 22:46:37.576445 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.576340 4376 scope.go:122] "RemoveContainer" containerID="87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a" Oct 29 22:46:37.576678 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:37.576655 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a\": container with ID starting with 87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a not found: ID does not exist" containerID="87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a" Oct 29 22:46:37.576779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.576687 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a"} err="failed to get container status \"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a\": rpc error: code = NotFound desc = could not find container \"87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a\": container with ID starting with 87b8ea1923796d7e70f73bf17afcf7ec6fc8e6a5b4891b095859944127d38b4a not found: ID does not exist" Oct 29 22:46:37.576779 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.576708 4376 scope.go:122] "RemoveContainer" containerID="83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a" Oct 29 22:46:37.585788 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.585769 4376 scope.go:122] "RemoveContainer" containerID="83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a" Oct 29 22:46:37.586076 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:37.586050 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a\": container with ID starting with 83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a not found: ID does not exist" containerID="83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a" Oct 29 22:46:37.586173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.586078 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a"} err="failed to get container status \"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a\": rpc error: code = NotFound desc = could not find container \"83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a\": container with ID starting with 83f14361d80f220b5facdbaddd6f544e927c73e39dd53857df97e27ab3af060a not found: ID does not exist" Oct 29 22:46:37.586173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.586093 4376 scope.go:122] "RemoveContainer" containerID="bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b" Oct 29 22:46:37.594600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.594581 4376 scope.go:122] "RemoveContainer" containerID="bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b" Oct 29 22:46:37.594890 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:37.594870 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b\": container with ID starting with bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b not found: ID does not exist" containerID="bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b" Oct 29 22:46:37.594989 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.594895 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b"} err="failed to get container status \"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b\": rpc error: code = NotFound desc = could not find container \"bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b\": container with ID starting with bf2e17399b020d89d053c78f61f0b3fffdb097dd9b748f36a1ee7d8e1e01617b not found: ID does not exist" Oct 29 22:46:37.625047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.625022 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 22:46:37.635618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.635595 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck"] Oct 29 22:46:37.670497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.670472 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:46:37.682384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.682346 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-580af2-58889c679-7xfnm"] Oct 29 22:46:37.717873 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.717848 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 22:46:37.726801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:37.726776 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/distinct-2-w0-f3b067-kserve-5bb94f7fc6-zqf6r"] Oct 29 22:46:38.285845 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:38.285799 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/distinct-w0-f3b067-kserve-56b74fb959-cfhck" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 29 22:46:39.271361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:39.271327 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" path="/var/lib/kubelet/pods/9a2fba5c-c2b7-44d1-87fd-12d623f8cc87/volumes" Oct 29 22:46:39.271846 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:39.271830 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" path="/var/lib/kubelet/pods/ee972f81-b171-40b5-8642-a8f34172e90d/volumes" Oct 29 22:46:39.272312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:39.272297 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" path="/var/lib/kubelet/pods/f2a90d29-40a0-4aa8-9991-61f234b3e273/volumes" Oct 29 22:46:46.433426 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:46.433385 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:46:47.950279 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:47.950244 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:46:47.950762 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:47.950501 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" containerID="cri-o://b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25" gracePeriod=30 Oct 29 22:46:47.956157 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:47.956130 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:46:47.956343 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:47.956308 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" containerID="cri-o://952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0" gracePeriod=30 Oct 29 22:46:49.089492 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.089455 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 22:46:49.090025 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.089743 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" podUID="30bc08dc-cb42-46aa-ac40-ba7c70701234" containerName="maas-api" containerID="cri-o://fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771" gracePeriod=30 Oct 29 22:46:49.371422 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.370830 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 22:46:49.402711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.402675 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\" (UniqueName: \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\") pod \"30bc08dc-cb42-46aa-ac40-ba7c70701234\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " Oct 29 22:46:49.402711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.402719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\") pod \"30bc08dc-cb42-46aa-ac40-ba7c70701234\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " Oct 29 22:46:49.403010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.402774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\") pod \"30bc08dc-cb42-46aa-ac40-ba7c70701234\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " Oct 29 22:46:49.403010 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.402799 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\") pod \"30bc08dc-cb42-46aa-ac40-ba7c70701234\" (UID: \"30bc08dc-cb42-46aa-ac40-ba7c70701234\") " Oct 29 22:46:49.403216 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.403190 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca" pod "30bc08dc-cb42-46aa-ac40-ba7c70701234" (UID: "30bc08dc-cb42-46aa-ac40-ba7c70701234"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:49.404962 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.404921 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls" pod "30bc08dc-cb42-46aa-ac40-ba7c70701234" (UID: "30bc08dc-cb42-46aa-ac40-ba7c70701234"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:49.405081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.404972 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs" pod "30bc08dc-cb42-46aa-ac40-ba7c70701234" (UID: "30bc08dc-cb42-46aa-ac40-ba7c70701234"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:49.405081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.405027 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk" pod "30bc08dc-cb42-46aa-ac40-ba7c70701234" (UID: "30bc08dc-cb42-46aa-ac40-ba7c70701234"). InnerVolumeSpecName "kube-api-access-hfwgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:49.504612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.504575 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwgk\" (UniqueName: \"kubernetes.io/projected/30bc08dc-cb42-46aa-ac40-ba7c70701234-kube-api-access-hfwgk\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.504612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.504609 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.504612 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.504618 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bc08dc-cb42-46aa-ac40-ba7c70701234-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.504860 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.504626 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bc08dc-cb42-46aa-ac40-ba7c70701234-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.689971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.689871 4376 generic.go:356] "Generic (PLEG): container finished" podID="30bc08dc-cb42-46aa-ac40-ba7c70701234" containerID="fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771" exitCode=0 Oct 29 22:46:49.689971 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.689953 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" Oct 29 22:46:49.690175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.689949 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" event={"ID":"30bc08dc-cb42-46aa-ac40-ba7c70701234","Type":"ContainerDied","Data":"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771"} Oct 29 22:46:49.690175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.690094 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v" event={"ID":"30bc08dc-cb42-46aa-ac40-ba7c70701234","Type":"ContainerDied","Data":"a82e6ca8fafa747bce9458fdc7362451afaa06cfb18d4771e0a9ad64dbf97bcc"} Oct 29 22:46:49.690175 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.690131 4376 scope.go:122] "RemoveContainer" containerID="fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771" Oct 29 22:46:49.700742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.700723 4376 scope.go:122] "RemoveContainer" containerID="fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771" Oct 29 22:46:49.701022 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:49.701000 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771\": container with ID starting with fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771 not found: ID does not exist" containerID="fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771" Oct 29 22:46:49.701092 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.701032 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771"} err="failed to get container status \"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771\": rpc error: code = NotFound desc = could not find container \"fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771\": container with ID starting with fa6e34aab0273e451e066835ab7cda7193f5dfe99ad5fbc262cd776a84adc771 not found: ID does not exist" Oct 29 22:46:49.745351 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.745298 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 22:46:49.758378 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:49.758347 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3b067-747d944689-msq2v"] Oct 29 22:46:51.272817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:51.272784 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bc08dc-cb42-46aa-ac40-ba7c70701234" path="/var/lib/kubelet/pods/30bc08dc-cb42-46aa-ac40-ba7c70701234/volumes" Oct 29 22:46:53.224202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.224138 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:46:53.248731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.248708 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:46:53.248995 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.248974 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249064 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249148 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249104 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249217 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249216 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.249272 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249242 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249330 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249362 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249402 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.249497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249413 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.249497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249420 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249497 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249484 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\") pod \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\" (UID: \"1360c02e-3a09-45ca-9d4c-f6241c9c66f0\") " Oct 29 22:46:53.249766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.249766 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.249722 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.250041 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.250021 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.250133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.250047 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.250133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.250061 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.250133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.250074 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.250133 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.250088 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.251158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.251138 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.251278 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.251264 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "kube-api-access-8cv7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.251950 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.251921 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:53.252037 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.251956 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token" pod "1360c02e-3a09-45ca-9d4c-f6241c9c66f0" (UID: "1360c02e-3a09-45ca-9d4c-f6241c9c66f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.350554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350503 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.350711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350593 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.350711 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350636 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.350863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350724 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.350960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350840 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.351072 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351045 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.351306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351107 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.351306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.350840 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.351306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351111 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.351306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351216 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.351306 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351249 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\") pod \"084fee8e-781d-417d-a76b-65b26b25c9d1\" (UID: \"084fee8e-781d-417d-a76b-65b26b25c9d1\") " Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351509 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351772 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351815 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351836 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084fee8e-781d-417d-a76b-65b26b25c9d1-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351850 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cv7h\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-kube-api-access-8cv7h\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351865 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351879 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351891 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1360c02e-3a09-45ca-9d4c-f6241c9c66f0-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.351960 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.351952 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.352838 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.352015 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.354225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.354187 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:53.354225 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.354214 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "kube-api-access-b9d8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.354357 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.354241 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.355708 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.355687 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy" pod "084fee8e-781d-417d-a76b-65b26b25c9d1" (UID: "084fee8e-781d-417d-a76b-65b26b25c9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.452827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452797 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.452827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452824 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.452827 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452834 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.453090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452842 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084fee8e-781d-417d-a76b-65b26b25c9d1-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.453090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452852 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9d8r\" (UniqueName: \"kubernetes.io/projected/084fee8e-781d-417d-a76b-65b26b25c9d1-kube-api-access-b9d8r\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.453090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.452861 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084fee8e-781d-417d-a76b-65b26b25c9d1-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.717081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.717044 4376 generic.go:356] "Generic (PLEG): container finished" podID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerID="b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25" exitCode=0 Oct 29 22:46:53.717271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.717122 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" Oct 29 22:46:53.717271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.717133 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" event={"ID":"1360c02e-3a09-45ca-9d4c-f6241c9c66f0","Type":"ContainerDied","Data":"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25"} Oct 29 22:46:53.717271 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.717205 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz" event={"ID":"1360c02e-3a09-45ca-9d4c-f6241c9c66f0","Type":"ContainerDied","Data":"11198de9e95a7589b4cb60748c9cfed2b41a50cd8ad19d098aa966301d5de926"} Oct 29 22:46:53.717477 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.717281 4376 scope.go:122] "RemoveContainer" containerID="b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25" Oct 29 22:46:53.718839 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.718813 4376 generic.go:356] "Generic (PLEG): container finished" podID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerID="952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0" exitCode=0 Oct 29 22:46:53.718982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.718853 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" event={"ID":"084fee8e-781d-417d-a76b-65b26b25c9d1","Type":"ContainerDied","Data":"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0"} Oct 29 22:46:53.718982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.718898 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" event={"ID":"084fee8e-781d-417d-a76b-65b26b25c9d1","Type":"ContainerDied","Data":"81a9fb859a3559e45f17be7b1389ebc515d5afc3ff0363a8f4e1211a15a2b7a2"} Oct 29 22:46:53.718982 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.718937 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl" Oct 29 22:46:53.728704 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.728487 4376 scope.go:122] "RemoveContainer" containerID="b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25" Oct 29 22:46:53.729367 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:53.729339 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25\": container with ID starting with b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25 not found: ID does not exist" containerID="b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25" Oct 29 22:46:53.729493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.729375 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25"} err="failed to get container status \"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25\": rpc error: code = NotFound desc = could not find container \"b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25\": container with ID starting with b9274d7a2d8d600d8a8e71fa1d94e9e89944105c1ced2e9cd8d2c3d2efe54f25 not found: ID does not exist" Oct 29 22:46:53.729493 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.729399 4376 scope.go:122] "RemoveContainer" containerID="952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0" Oct 29 22:46:53.737859 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.737841 4376 scope.go:122] "RemoveContainer" containerID="952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0" Oct 29 22:46:53.738113 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:46:53.738095 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0\": container with ID starting with 952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0 not found: ID does not exist" containerID="952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0" Oct 29 22:46:53.738190 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.738118 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0"} err="failed to get container status \"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0\": rpc error: code = NotFound desc = could not find container \"952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0\": container with ID starting with 952fc5b3a49b54ec809f90356b6278f7dee4f013acffebe826513fccdfe6b7a0 not found: ID does not exist" Oct 29 22:46:53.775166 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.775144 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:46:53.787534 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.787512 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-j7jnl"] Oct 29 22:46:53.816719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.816695 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:46:53.822473 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:53.822448 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-c7hgz"] Oct 29 22:46:55.271458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:55.271421 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" path="/var/lib/kubelet/pods/084fee8e-781d-417d-a76b-65b26b25c9d1/volumes" Oct 29 22:46:55.271876 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:46:55.271861 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" path="/var/lib/kubelet/pods/1360c02e-3a09-45ca-9d4c-f6241c9c66f0/volumes" Oct 29 22:47:04.625346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.625305 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:47:04.627017 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.626955 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" podUID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" containerName="maas-api" containerID="cri-o://5c7dc8665b51f142d5c561f337c398925f6b41957d5de3552d09b385af6a64c2" gracePeriod=30 Oct 29 22:47:04.787629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.787594 4376 generic.go:356] "Generic (PLEG): container finished" podID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" containerID="5c7dc8665b51f142d5c561f337c398925f6b41957d5de3552d09b385af6a64c2" exitCode=0 Oct 29 22:47:04.787800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.787671 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" event={"ID":"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96","Type":"ContainerDied","Data":"5c7dc8665b51f142d5c561f337c398925f6b41957d5de3552d09b385af6a64c2"} Oct 29 22:47:04.913966 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.913939 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:47:04.984446 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.984369 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\") pod \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " Oct 29 22:47:04.984831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.984807 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\") pod \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " Oct 29 22:47:04.985369 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.985349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\" (UniqueName: \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\") pod \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " Oct 29 22:47:04.985690 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.985545 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\") pod \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\" (UID: \"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96\") " Oct 29 22:47:04.986436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.986134 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca" pod "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" (UID: "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:04.986436 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.986403 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:04.988649 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.988625 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:47:04.989135 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:04.989103 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" podUID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" containerName="maas-api" containerID="cri-o://b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb" gracePeriod=30 Oct 29 22:47:05.005461 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.005428 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs" pod "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" (UID: "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:05.005742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.005717 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls" pod "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" (UID: "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:05.007371 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.007347 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l" pod "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" (UID: "b3cbeffd-8427-4f7b-9c9e-8394f6a12a96"). InnerVolumeSpecName "kube-api-access-w6c7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:05.087629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.087591 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6c7l\" (UniqueName: \"kubernetes.io/projected/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-kube-api-access-w6c7l\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.087629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.087628 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.087629 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.087638 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.243506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.243478 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:47:05.391016 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.390978 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\" (UniqueName: \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\") pod \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " Oct 29 22:47:05.391212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.391084 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\") pod \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " Oct 29 22:47:05.391212 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.391185 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\") pod \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " Oct 29 22:47:05.391346 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.391213 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\") pod \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\" (UID: \"814c9c42-6fba-4eea-887a-8d4acc7c26a9\") " Oct 29 22:47:05.391610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.391584 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca" pod "814c9c42-6fba-4eea-887a-8d4acc7c26a9" (UID: "814c9c42-6fba-4eea-887a-8d4acc7c26a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:05.393312 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.393279 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls" pod "814c9c42-6fba-4eea-887a-8d4acc7c26a9" (UID: "814c9c42-6fba-4eea-887a-8d4acc7c26a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:05.393488 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.393457 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv" pod "814c9c42-6fba-4eea-887a-8d4acc7c26a9" (UID: "814c9c42-6fba-4eea-887a-8d4acc7c26a9"). InnerVolumeSpecName "kube-api-access-jjhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:05.393620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.393560 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs" pod "814c9c42-6fba-4eea-887a-8d4acc7c26a9" (UID: "814c9c42-6fba-4eea-887a-8d4acc7c26a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:05.493073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.493038 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.493073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.493068 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/814c9c42-6fba-4eea-887a-8d4acc7c26a9-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.493073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.493077 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814c9c42-6fba-4eea-887a-8d4acc7c26a9-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.493324 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.493087 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhbv\" (UniqueName: \"kubernetes.io/projected/814c9c42-6fba-4eea-887a-8d4acc7c26a9-kube-api-access-jjhbv\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:05.794874 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.794785 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" Oct 29 22:47:05.795355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.794793 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf" event={"ID":"b3cbeffd-8427-4f7b-9c9e-8394f6a12a96","Type":"ContainerDied","Data":"3642e1295731e1da0f66ac4aee92e90dfd49a1899cea9a2f19dd28ee3d2f766a"} Oct 29 22:47:05.795355 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.794976 4376 scope.go:122] "RemoveContainer" containerID="5c7dc8665b51f142d5c561f337c398925f6b41957d5de3552d09b385af6a64c2" Oct 29 22:47:05.796480 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.796426 4376 generic.go:356] "Generic (PLEG): container finished" podID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" containerID="b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb" exitCode=0 Oct 29 22:47:05.796571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.796479 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" event={"ID":"814c9c42-6fba-4eea-887a-8d4acc7c26a9","Type":"ContainerDied","Data":"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb"} Oct 29 22:47:05.796571 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.796514 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" event={"ID":"814c9c42-6fba-4eea-887a-8d4acc7c26a9","Type":"ContainerDied","Data":"925ae324a60e1bc5634e730c16901cc3d381208dae35686b14994a702a5a5a2b"} Oct 29 22:47:05.796683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.796575 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d" Oct 29 22:47:05.816228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.816203 4376 scope.go:122] "RemoveContainer" containerID="b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb" Oct 29 22:47:05.825765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.825741 4376 scope.go:122] "RemoveContainer" containerID="b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb" Oct 29 22:47:05.826102 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:05.826074 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb\": container with ID starting with b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb not found: ID does not exist" containerID="b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb" Oct 29 22:47:05.826164 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.826112 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb"} err="failed to get container status \"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb\": rpc error: code = NotFound desc = could not find container \"b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb\": container with ID starting with b94f45e0bf036df156080b9be3e59e84634f4c747916ba1046e7cd7ae65adbcb not found: ID does not exist" Oct 29 22:47:05.868112 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.868082 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:47:05.873387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.873360 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-749f72-5664947bd7-vbndf"] Oct 29 22:47:05.915073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.914949 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:47:05.920384 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:05.920357 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-07e166-6b6689df74-xvf4d"] Oct 29 22:47:07.270618 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:07.270581 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" path="/var/lib/kubelet/pods/814c9c42-6fba-4eea-887a-8d4acc7c26a9/volumes" Oct 29 22:47:07.271084 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:07.271068 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" path="/var/lib/kubelet/pods/b3cbeffd-8427-4f7b-9c9e-8394f6a12a96/volumes" Oct 29 22:47:14.303192 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:14.303154 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:47:14.303636 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:14.303401 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" containerID="cri-o://96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d" gracePeriod=30 Oct 29 22:47:14.472364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:14.472326 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:47:14.472640 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:14.472606 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" containerID="cri-o://250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257" gracePeriod=30 Oct 29 22:47:16.434062 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:16.434022 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:47:19.602249 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.602215 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:47:19.660809 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.660772 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.661054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.660834 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.661054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.660893 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.661054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.660957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.661054 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.660999 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.662568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.661084 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.662568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.661144 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.662568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.661183 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.662568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.661238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\") pod \"48adcf8c-be0d-487e-a79a-a4cf4e066126\" (UID: \"48adcf8c-be0d-487e-a79a-a4cf4e066126\") " Oct 29 22:47:19.662568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.662456 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:19.663070 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.662727 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.665104 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.665075 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.669287 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.669259 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.670129 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.670103 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.670348 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.670324 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.670614 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.670596 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.672947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.672504 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:19.672947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.672764 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr" pod "48adcf8c-be0d-487e-a79a-a4cf4e066126" (UID: "48adcf8c-be0d-487e-a79a-a4cf4e066126"). InnerVolumeSpecName "kube-api-access-zb7mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.727631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.727595 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:47:19.762680 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762642 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.762887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762693 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.762887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762729 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.762887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762769 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.762887 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762794 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.763159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.762973 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.763159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763046 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.763159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763081 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.763159 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\") pod \"c9d2bc07-2df6-4758-b256-995e638df935\" (UID: \"c9d2bc07-2df6-4758-b256-995e638df935\") " Oct 29 22:47:19.763390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763176 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.763390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763187 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.763390 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763268 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:19.763554 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.763632 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763610 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763645 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763659 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763673 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763686 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763699 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb7mr\" (UniqueName: \"kubernetes.io/projected/48adcf8c-be0d-487e-a79a-a4cf4e066126-kube-api-access-zb7mr\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763715 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763727 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763741 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48adcf8c-be0d-487e-a79a-a4cf4e066126-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763755 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763768 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d2bc07-2df6-4758-b256-995e638df935-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763610 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763782 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.763831 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.763796 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48adcf8c-be0d-487e-a79a-a4cf4e066126-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.765170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.765138 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.765170 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.765151 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:19.765505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.765487 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:19.765570 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.765526 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf" pod "c9d2bc07-2df6-4758-b256-995e638df935" (UID: "c9d2bc07-2df6-4758-b256-995e638df935"). InnerVolumeSpecName "kube-api-access-gzmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.865362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.865277 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.865362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.865306 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.865362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.865316 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmqf\" (UniqueName: \"kubernetes.io/projected/c9d2bc07-2df6-4758-b256-995e638df935-kube-api-access-gzmqf\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.865362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.865326 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d2bc07-2df6-4758-b256-995e638df935-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.865362 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.865335 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d2bc07-2df6-4758-b256-995e638df935-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.899847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.899808 4376 generic.go:356] "Generic (PLEG): container finished" podID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerID="96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d" exitCode=0 Oct 29 22:47:19.900078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.899873 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" Oct 29 22:47:19.900078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.899895 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerDied","Data":"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d"} Oct 29 22:47:19.900078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.900014 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp" event={"ID":"48adcf8c-be0d-487e-a79a-a4cf4e066126","Type":"ContainerDied","Data":"840ea6c982bcd67f61f22527b935430a079daca3f02fad210ab8c40b0f4e5c5a"} Oct 29 22:47:19.900078 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.900044 4376 scope.go:122] "RemoveContainer" containerID="96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d" Oct 29 22:47:19.901974 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.901947 4376 generic.go:356] "Generic (PLEG): container finished" podID="c9d2bc07-2df6-4758-b256-995e638df935" containerID="250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257" exitCode=0 Oct 29 22:47:19.902096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.902007 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerDied","Data":"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257"} Oct 29 22:47:19.902096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.902053 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" event={"ID":"c9d2bc07-2df6-4758-b256-995e638df935","Type":"ContainerDied","Data":"3ac7a542a8aeeb61cc52d55d090b9195bcacd47afda69922cabcd9a662f17bb2"} Oct 29 22:47:19.902096 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.902054 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz" Oct 29 22:47:19.910867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.910842 4376 scope.go:122] "RemoveContainer" containerID="173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e" Oct 29 22:47:19.920019 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.919998 4376 scope.go:122] "RemoveContainer" containerID="96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d" Oct 29 22:47:19.920294 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:19.920273 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d\": container with ID starting with 96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d not found: ID does not exist" containerID="96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d" Oct 29 22:47:19.920339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.920305 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d"} err="failed to get container status \"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d\": rpc error: code = NotFound desc = could not find container \"96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d\": container with ID starting with 96f4c372eefcc2196a1edb19dff49a56c5392b533b442b7d2107125a0a7c829d not found: ID does not exist" Oct 29 22:47:19.920339 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.920324 4376 scope.go:122] "RemoveContainer" containerID="173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e" Oct 29 22:47:19.920552 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:19.920534 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e\": container with ID starting with 173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e not found: ID does not exist" containerID="173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e" Oct 29 22:47:19.920660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.920552 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e"} err="failed to get container status \"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e\": rpc error: code = NotFound desc = could not find container \"173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e\": container with ID starting with 173afcb79dfa71c4f555265f99c7fdc8bf1c709804a1809647b3a773d4dab53e not found: ID does not exist" Oct 29 22:47:19.920660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.920565 4376 scope.go:122] "RemoveContainer" containerID="250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257" Oct 29 22:47:19.929195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.929177 4376 scope.go:122] "RemoveContainer" containerID="c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1" Oct 29 22:47:19.940046 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.940020 4376 scope.go:122] "RemoveContainer" containerID="250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257" Oct 29 22:47:19.940341 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:19.940318 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257\": container with ID starting with 250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257 not found: ID does not exist" containerID="250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257" Oct 29 22:47:19.940412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.940350 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257"} err="failed to get container status \"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257\": rpc error: code = NotFound desc = could not find container \"250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257\": container with ID starting with 250d236bb2c8b4875c99a86953fb1c57e2826877786484f6221ab0c59cdfd257 not found: ID does not exist" Oct 29 22:47:19.940412 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.940369 4376 scope.go:122] "RemoveContainer" containerID="c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1" Oct 29 22:47:19.940581 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:19.940564 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1\": container with ID starting with c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1 not found: ID does not exist" containerID="c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1" Oct 29 22:47:19.940622 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.940587 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1"} err="failed to get container status \"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1\": rpc error: code = NotFound desc = could not find container \"c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1\": container with ID starting with c541a51cf1d703ef856f92c8faa8b7204cd06c662acae6fc621005b712cb22c1 not found: ID does not exist" Oct 29 22:47:19.957013 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.956969 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:47:19.961660 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.961632 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-ttkmz"] Oct 29 22:47:19.978631 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.978598 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 22:47:19.985803 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:19.985772 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 22:47:20.003643 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:20.003583 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:47:20.016441 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:20.016409 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-749f72-openshift-default-5f5fffc597-97gbp"] Oct 29 22:47:21.270462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:21.270429 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" path="/var/lib/kubelet/pods/48adcf8c-be0d-487e-a79a-a4cf4e066126/volumes" Oct 29 22:47:21.270957 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:21.270941 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d2bc07-2df6-4758-b256-995e638df935" path="/var/lib/kubelet/pods/c9d2bc07-2df6-4758-b256-995e638df935/volumes" Oct 29 22:47:25.659460 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:25.659424 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:47:25.660211 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:25.659737 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" podUID="72f864ce-c860-4d20-b75f-35b4bd13cee7" containerName="maas-api" containerID="cri-o://3692ba22ecfe8a9726612ebf9d6d6c84243fbe6022d980b0b51cda3a8e6c2503" gracePeriod=30 Oct 29 22:47:25.937605 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:25.937571 4376 generic.go:356] "Generic (PLEG): container finished" podID="72f864ce-c860-4d20-b75f-35b4bd13cee7" containerID="3692ba22ecfe8a9726612ebf9d6d6c84243fbe6022d980b0b51cda3a8e6c2503" exitCode=0 Oct 29 22:47:25.937765 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:25.937642 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" event={"ID":"72f864ce-c860-4d20-b75f-35b4bd13cee7","Type":"ContainerDied","Data":"3692ba22ecfe8a9726612ebf9d6d6c84243fbe6022d980b0b51cda3a8e6c2503"} Oct 29 22:47:25.963334 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:25.963307 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:47:26.035082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.035024 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\") pod \"72f864ce-c860-4d20-b75f-35b4bd13cee7\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " Oct 29 22:47:26.035082 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.035087 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\") pod \"72f864ce-c860-4d20-b75f-35b4bd13cee7\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " Oct 29 22:47:26.035382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.035217 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\") pod \"72f864ce-c860-4d20-b75f-35b4bd13cee7\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " Oct 29 22:47:26.035382 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.035326 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\" (UniqueName: \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\") pod \"72f864ce-c860-4d20-b75f-35b4bd13cee7\" (UID: \"72f864ce-c860-4d20-b75f-35b4bd13cee7\") " Oct 29 22:47:26.035635 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.035600 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca" pod "72f864ce-c860-4d20-b75f-35b4bd13cee7" (UID: "72f864ce-c860-4d20-b75f-35b4bd13cee7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:26.037376 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.037354 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls" pod "72f864ce-c860-4d20-b75f-35b4bd13cee7" (UID: "72f864ce-c860-4d20-b75f-35b4bd13cee7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:26.037505 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.037439 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86" pod "72f864ce-c860-4d20-b75f-35b4bd13cee7" (UID: "72f864ce-c860-4d20-b75f-35b4bd13cee7"). InnerVolumeSpecName "kube-api-access-n5h86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:26.037742 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.037727 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs" pod "72f864ce-c860-4d20-b75f-35b4bd13cee7" (UID: "72f864ce-c860-4d20-b75f-35b4bd13cee7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:26.136847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.136805 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5h86\" (UniqueName: \"kubernetes.io/projected/72f864ce-c860-4d20-b75f-35b4bd13cee7-kube-api-access-n5h86\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:26.136847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.136840 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:26.136847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.136849 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f864ce-c860-4d20-b75f-35b4bd13cee7-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:26.136847 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.136857 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f864ce-c860-4d20-b75f-35b4bd13cee7-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:26.944431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.944398 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" event={"ID":"72f864ce-c860-4d20-b75f-35b4bd13cee7","Type":"ContainerDied","Data":"1cdc06c10c452962495f7aea1acf4516e60b9aa5a76c37a2e15a718c5563b4b2"} Oct 29 22:47:26.944431 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.944425 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg" Oct 29 22:47:26.944997 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:26.944459 4376 scope.go:122] "RemoveContainer" containerID="3692ba22ecfe8a9726612ebf9d6d6c84243fbe6022d980b0b51cda3a8e6c2503" Oct 29 22:47:27.001583 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:27.001553 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:47:27.015221 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:27.015189 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-201a3d-564876bf58-kdzrg"] Oct 29 22:47:27.270955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:27.270893 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f864ce-c860-4d20-b75f-35b4bd13cee7" path="/var/lib/kubelet/pods/72f864ce-c860-4d20-b75f-35b4bd13cee7/volumes" Oct 29 22:47:34.999878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:34.999774 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" containerID="cri-o://dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76" gracePeriod=45 Oct 29 22:47:35.009283 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.009251 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" containerID="cri-o://6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3" gracePeriod=45 Oct 29 22:47:35.022939 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:35.022874 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c364880_627c_428e_8713_6d29733874bd.slice/crio-dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:47:35.265752 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.265678 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:47:35.298564 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.298526 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:47:35.334160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334128 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334200 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334232 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\" (UniqueName: \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.334501 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334470 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334610 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334525 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\") pod \"7c364880-627c-428e-8713-6d29733874bd\" (UID: \"7c364880-627c-428e-8713-6d29733874bd\") " Oct 29 22:47:35.334662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334530 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.334836 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.334814 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.335624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.335339 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.335624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.335366 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.335624 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.335379 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.336936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.336913 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:35.337085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.337065 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.337145 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.337088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8" pod "7c364880-627c-428e-8713-6d29733874bd" (UID: "7c364880-627c-428e-8713-6d29733874bd"). InnerVolumeSpecName "kube-api-access-nnmj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:35.435731 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435693 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.435937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435752 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.435937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435808 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.435937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435832 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\" (UniqueName: \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.435937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435927 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.436184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.435995 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\") pod \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\" (UID: \"4e69a718-86ef-4ea7-a475-fc5e7fe62476\") " Oct 29 22:47:35.436184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436047 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.436184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.436341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436260 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.436484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436465 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnmj8\" (UniqueName: \"kubernetes.io/projected/7c364880-627c-428e-8713-6d29733874bd-kube-api-access-nnmj8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.436484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436485 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-model-cache\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.436484 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436495 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c364880-627c-428e-8713-6d29733874bd-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.436714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436503 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c364880-627c-428e-8713-6d29733874bd-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.436714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436512 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-home\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.436714 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.436521 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tmp-dir\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.438103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.438080 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "kube-api-access-tjl55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:35.438103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.438084 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.438244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.438120 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs" pod "4e69a718-86ef-4ea7-a475-fc5e7fe62476" (UID: "4e69a718-86ef-4ea7-a475-fc5e7fe62476"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:35.537478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.537382 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e69a718-86ef-4ea7-a475-fc5e7fe62476-dshm\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.537478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.537413 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e69a718-86ef-4ea7-a475-fc5e7fe62476-tls-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.537478 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.537424 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjl55\" (UniqueName: \"kubernetes.io/projected/4e69a718-86ef-4ea7-a475-fc5e7fe62476-kube-api-access-tjl55\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.996162 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.996126 4376 generic.go:356] "Generic (PLEG): container finished" podID="7c364880-627c-428e-8713-6d29733874bd" containerID="dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76" exitCode=0 Oct 29 22:47:35.996364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.996174 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" Oct 29 22:47:35.996364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.996208 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" event={"ID":"7c364880-627c-428e-8713-6d29733874bd","Type":"ContainerDied","Data":"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76"} Oct 29 22:47:35.996364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.996250 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7" event={"ID":"7c364880-627c-428e-8713-6d29733874bd","Type":"ContainerDied","Data":"df1c7fbf0a5ac28ca2ad249e1e648644c19473366310a583bd31bb589c74289a"} Oct 29 22:47:35.996364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.996272 4376 scope.go:122] "RemoveContainer" containerID="dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76" Oct 29 22:47:35.997687 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.997659 4376 generic.go:356] "Generic (PLEG): container finished" podID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerID="6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3" exitCode=0 Oct 29 22:47:35.997817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.997686 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" event={"ID":"4e69a718-86ef-4ea7-a475-fc5e7fe62476","Type":"ContainerDied","Data":"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3"} Oct 29 22:47:35.997817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.997709 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" Oct 29 22:47:35.997817 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:35.997725 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s" event={"ID":"4e69a718-86ef-4ea7-a475-fc5e7fe62476","Type":"ContainerDied","Data":"bcf86f1ec9067f912722a2532b14dc5835a69ed99446df25c4a0c44d2f9d1438"} Oct 29 22:47:36.005758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.005655 4376 scope.go:122] "RemoveContainer" containerID="dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76" Oct 29 22:47:36.006041 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:36.006015 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76\": container with ID starting with dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76 not found: ID does not exist" containerID="dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76" Oct 29 22:47:36.006087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.006047 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76"} err="failed to get container status \"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76\": rpc error: code = NotFound desc = could not find container \"dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76\": container with ID starting with dd2b10c8bbb4df94bc0718a0868a7091f446619876308421f21a7e2dbc12ee76 not found: ID does not exist" Oct 29 22:47:36.006087 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.006065 4376 scope.go:122] "RemoveContainer" containerID="6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3" Oct 29 22:47:36.024122 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.024098 4376 scope.go:122] "RemoveContainer" containerID="6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3" Oct 29 22:47:36.024495 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:36.024473 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3\": container with ID starting with 6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3 not found: ID does not exist" containerID="6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3" Oct 29 22:47:36.024544 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.024505 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3"} err="failed to get container status \"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3\": rpc error: code = NotFound desc = could not find container \"6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3\": container with ID starting with 6a57944917adb3769152eb9b38c3a0914efa979917354cf28b534189bce370c3 not found: ID does not exist" Oct 29 22:47:36.069598 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.069554 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 22:47:36.082001 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.081973 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/premium-w6-863f1b-kserve-79bc7f5bbd-9p28s"] Oct 29 22:47:36.120568 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.120529 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 22:47:36.123509 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:36.123479 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-w6-863f1b-kserve-8676d45f59-9gqm7"] Oct 29 22:47:37.270955 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:37.270885 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" path="/var/lib/kubelet/pods/4e69a718-86ef-4ea7-a475-fc5e7fe62476/volumes" Oct 29 22:47:37.271370 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:37.271346 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c364880-627c-428e-8713-6d29733874bd" path="/var/lib/kubelet/pods/7c364880-627c-428e-8713-6d29733874bd/volumes" Oct 29 22:47:46.433880 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:46.433834 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:47:51.784539 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:51.784493 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:47:51.785090 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:51.784768 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" podUID="a9fff7a2-f346-408c-83ee-b8a591643a9f" containerName="maas-api" containerID="cri-o://fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c" gracePeriod=30 Oct 29 22:47:52.051947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.051866 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:47:52.085670 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.085635 4376 generic.go:356] "Generic (PLEG): container finished" podID="a9fff7a2-f346-408c-83ee-b8a591643a9f" containerID="fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c" exitCode=0 Oct 29 22:47:52.085854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.085689 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" event={"ID":"a9fff7a2-f346-408c-83ee-b8a591643a9f","Type":"ContainerDied","Data":"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c"} Oct 29 22:47:52.085854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.085728 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" event={"ID":"a9fff7a2-f346-408c-83ee-b8a591643a9f","Type":"ContainerDied","Data":"00e0f5a0c9cf9ef5f209642059695465ae6fe82dd95286f3908b9821cebd5691"} Oct 29 22:47:52.085854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.085749 4376 scope.go:122] "RemoveContainer" containerID="fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c" Oct 29 22:47:52.085854 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.085690 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l" Oct 29 22:47:52.094720 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.094692 4376 scope.go:122] "RemoveContainer" containerID="fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c" Oct 29 22:47:52.095230 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:47:52.095083 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c\": container with ID starting with fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c not found: ID does not exist" containerID="fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c" Oct 29 22:47:52.095230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.095124 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c"} err="failed to get container status \"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c\": rpc error: code = NotFound desc = could not find container \"fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c\": container with ID starting with fe99277567a39bafa95890b3f85eb7285766021339522113d385ca55ff23356c not found: ID does not exist" Oct 29 22:47:52.111450 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.111417 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\") pod \"a9fff7a2-f346-408c-83ee-b8a591643a9f\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " Oct 29 22:47:52.111597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.111456 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\") pod \"a9fff7a2-f346-408c-83ee-b8a591643a9f\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " Oct 29 22:47:52.111597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.111541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\") pod \"a9fff7a2-f346-408c-83ee-b8a591643a9f\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " Oct 29 22:47:52.111707 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.111684 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\" (UniqueName: \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\") pod \"a9fff7a2-f346-408c-83ee-b8a591643a9f\" (UID: \"a9fff7a2-f346-408c-83ee-b8a591643a9f\") " Oct 29 22:47:52.111820 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.111791 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca" pod "a9fff7a2-f346-408c-83ee-b8a591643a9f" (UID: "a9fff7a2-f346-408c-83ee-b8a591643a9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:52.112098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.112069 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fff7a2-f346-408c-83ee-b8a591643a9f-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:52.113746 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.113719 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs" pod "a9fff7a2-f346-408c-83ee-b8a591643a9f" (UID: "a9fff7a2-f346-408c-83ee-b8a591643a9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:52.113897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.113875 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8" pod "a9fff7a2-f346-408c-83ee-b8a591643a9f" (UID: "a9fff7a2-f346-408c-83ee-b8a591643a9f"). InnerVolumeSpecName "kube-api-access-9wgr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:52.113999 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.113980 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls" pod "a9fff7a2-f346-408c-83ee-b8a591643a9f" (UID: "a9fff7a2-f346-408c-83ee-b8a591643a9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:52.213388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.213352 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgr8\" (UniqueName: \"kubernetes.io/projected/a9fff7a2-f346-408c-83ee-b8a591643a9f-kube-api-access-9wgr8\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:52.213388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.213384 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:52.213388 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.213394 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fff7a2-f346-408c-83ee-b8a591643a9f-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:52.436665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.436626 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:47:52.443403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:52.443362 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-1e74c3-db586d65b-6kl4l"] Oct 29 22:47:53.270401 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:47:53.270360 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fff7a2-f346-408c-83ee-b8a591643a9f" path="/var/lib/kubelet/pods/a9fff7a2-f346-408c-83ee-b8a591643a9f/volumes" Oct 29 22:48:01.614531 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:01.614492 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:48:01.615210 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:01.614721 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" containerID="cri-o://e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300" gracePeriod=30 Oct 29 22:48:06.876855 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.876828 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:48:06.972816 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.972775 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.972829 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.972966 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973043 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973025 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973077 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973121 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973156 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973191 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973245 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973216 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\") pod \"553a8d1c-4d61-4341-a642-699027f3f480\" (UID: \"553a8d1c-4d61-4341-a642-699027f3f480\") " Oct 29 22:48:06.973528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:06.973528 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973454 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:06.973683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973460 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:06.973683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973649 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:06.973683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973670 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:06.973871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973678 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:06.973871 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.973699 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:06.975228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.975188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:06.975380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.975362 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "kube-api-access-s5h8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:06.975471 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.975450 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:06.975878 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:06.975844 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo" pod "553a8d1c-4d61-4341-a642-699027f3f480" (UID: "553a8d1c-4d61-4341-a642-699027f3f480"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:07.075103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075063 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075098 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5h8s\" (UniqueName: \"kubernetes.io/projected/553a8d1c-4d61-4341-a642-699027f3f480-kube-api-access-s5h8s\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075113 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553a8d1c-4d61-4341-a642-699027f3f480-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075125 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075136 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553a8d1c-4d61-4341-a642-699027f3f480-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075147 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.075340 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.075158 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553a8d1c-4d61-4341-a642-699027f3f480-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:07.162140 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.162103 4376 generic.go:356] "Generic (PLEG): container finished" podID="553a8d1c-4d61-4341-a642-699027f3f480" containerID="e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300" exitCode=0 Oct 29 22:48:07.162316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.162152 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" Oct 29 22:48:07.162316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.162181 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerDied","Data":"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300"} Oct 29 22:48:07.162316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.162239 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz" event={"ID":"553a8d1c-4d61-4341-a642-699027f3f480","Type":"ContainerDied","Data":"6070ea76f02246874f4879e4840fa2e127ccae72ff411182cde272ab9d1ad225"} Oct 29 22:48:07.162316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.162261 4376 scope.go:122] "RemoveContainer" containerID="e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300" Oct 29 22:48:07.172246 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.172231 4376 scope.go:122] "RemoveContainer" containerID="3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e" Oct 29 22:48:07.180647 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.180630 4376 scope.go:122] "RemoveContainer" containerID="e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300" Oct 29 22:48:07.180890 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:07.180871 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300\": container with ID starting with e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300 not found: ID does not exist" containerID="e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300" Oct 29 22:48:07.180961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.180899 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300"} err="failed to get container status \"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300\": rpc error: code = NotFound desc = could not find container \"e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300\": container with ID starting with e83a6aeb739dff4beb913dc7b7b4236d7456791c8c5802485896b1973cdce300 not found: ID does not exist" Oct 29 22:48:07.180961 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.180931 4376 scope.go:122] "RemoveContainer" containerID="3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e" Oct 29 22:48:07.181175 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:07.181156 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e\": container with ID starting with 3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e not found: ID does not exist" containerID="3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e" Oct 29 22:48:07.181241 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.181184 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e"} err="failed to get container status \"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e\": rpc error: code = NotFound desc = could not find container \"3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e\": container with ID starting with 3976bab3514e5fedf33573b8df9c5ca9dfd0791cb3ffa8a537649554b92cf87e not found: ID does not exist" Oct 29 22:48:07.221888 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.221841 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:48:07.232454 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.232393 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-wwsxz"] Oct 29 22:48:07.270314 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:07.270279 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553a8d1c-4d61-4341-a642-699027f3f480" path="/var/lib/kubelet/pods/553a8d1c-4d61-4341-a642-699027f3f480/volumes" Oct 29 22:48:12.671898 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:12.671859 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:48:16.433921 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:16.433866 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:48:17.672423 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:17.672375 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" containerID="cri-o://33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c" gracePeriod=25 Oct 29 22:48:21.721049 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:21.721004 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:31.721171 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:31.721125 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:37.923940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.923897 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:48:37.992222 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.992183 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\") pod \"8723965e-5757-49b5-a066-865ce2b12406\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " Oct 29 22:48:37.992403 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.992342 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\" (UniqueName: \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\") pod \"8723965e-5757-49b5-a066-865ce2b12406\" (UID: \"8723965e-5757-49b5-a066-865ce2b12406\") " Oct 29 22:48:37.992589 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.992565 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume" pod "8723965e-5757-49b5-a066-865ce2b12406" (UID: "8723965e-5757-49b5-a066-865ce2b12406"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:37.992717 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.992703 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8723965e-5757-49b5-a066-865ce2b12406-plugins-config-volume\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:37.994597 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:37.994570 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m" pod "8723965e-5757-49b5-a066-865ce2b12406" (UID: "8723965e-5757-49b5-a066-865ce2b12406"). InnerVolumeSpecName "kube-api-access-sd29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:38.094295 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.094259 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd29m\" (UniqueName: \"kubernetes.io/projected/8723965e-5757-49b5-a066-865ce2b12406-kube-api-access-sd29m\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.314994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.314883 4376 generic.go:356] "Generic (PLEG): container finished" podID="8723965e-5757-49b5-a066-865ce2b12406" containerID="33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c" exitCode=0 Oct 29 22:48:38.314994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.314943 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" Oct 29 22:48:38.314994 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.314975 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" event={"ID":"8723965e-5757-49b5-a066-865ce2b12406","Type":"ContainerDied","Data":"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c"} Oct 29 22:48:38.315344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.315025 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84" event={"ID":"8723965e-5757-49b5-a066-865ce2b12406","Type":"ContainerDied","Data":"9dcdf8f8cf06e3fd26e31d409512bbc105bcc2d9906aea54bc6784d59eb64473"} Oct 29 22:48:38.315344 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.315047 4376 scope.go:122] "RemoveContainer" containerID="33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c" Oct 29 22:48:38.324146 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.324129 4376 scope.go:122] "RemoveContainer" containerID="33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c" Oct 29 22:48:38.324417 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:38.324398 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c\": container with ID starting with 33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c not found: ID does not exist" containerID="33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c" Oct 29 22:48:38.324466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.324428 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c"} err="failed to get container status \"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c\": rpc error: code = NotFound desc = could not find container \"33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c\": container with ID starting with 33f2ca844313ec43d332f4ff93180ebfa894e9633579229a474c29b2fd2c4d7c not found: ID does not exist" Oct 29 22:48:38.384936 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.384885 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:48:38.398310 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:38.398280 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8d569e-59fc4d8578-b5b84"] Oct 29 22:48:39.271061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:39.271026 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8723965e-5757-49b5-a066-865ce2b12406" path="/var/lib/kubelet/pods/8723965e-5757-49b5-a066-865ce2b12406/volumes" Oct 29 22:48:39.679884 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:39.679796 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 22:48:39.680085 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:39.680060 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" podUID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" containerName="discovery" containerID="cri-o://ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d" gracePeriod=30 Oct 29 22:48:39.939600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:39.939534 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 22:48:40.013958 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.013914 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014067 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014103 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014132 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014244 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014364 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014330 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\") pod \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\" (UID: \"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c\") " Oct 29 22:48:40.014985 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.014949 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:40.016732 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.016670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "kube-api-access-zk2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:40.016872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.016780 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:40.016872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.016792 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:40.016872 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.016812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:40.017097 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.017078 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:40.017228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.017209 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs" pod "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" (UID: "89b0c9d7-f5e3-408a-ac68-92b6a9a8387c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:40.115240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115208 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2f6\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-kube-api-access-zk2f6\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115236 4376 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-local-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115240 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115246 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115254 4376 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-kubeconfig\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115262 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-ca-configmap\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115273 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-istio-csr-dns-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.115496 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.115283 4376 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c-cacerts\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:40.326621 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.326585 4376 generic.go:356] "Generic (PLEG): container finished" podID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" containerID="ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d" exitCode=0 Oct 29 22:48:40.327137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.326634 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" Oct 29 22:48:40.327137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.326664 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" event={"ID":"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c","Type":"ContainerDied","Data":"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d"} Oct 29 22:48:40.327137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.326718 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld" event={"ID":"89b0c9d7-f5e3-408a-ac68-92b6a9a8387c","Type":"ContainerDied","Data":"2e1f37fc564b0f51f83c5164cea60837f7991f3540fedeb166e6202c6c4c4291"} Oct 29 22:48:40.327137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.326747 4376 scope.go:122] "RemoveContainer" containerID="ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d" Oct 29 22:48:40.344202 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.344177 4376 scope.go:122] "RemoveContainer" containerID="ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d" Oct 29 22:48:40.344492 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:40.344471 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d\": container with ID starting with ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d not found: ID does not exist" containerID="ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d" Oct 29 22:48:40.344563 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.344501 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d"} err="failed to get container status \"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d\": rpc error: code = NotFound desc = could not find container \"ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d\": container with ID starting with ee7eca2fa1e4f5bd184eb0c403ebbf7c68c64b0970aebdb974ffeaba7b3fe96d not found: ID does not exist" Oct 29 22:48:40.389380 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.389354 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 22:48:40.402865 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:40.402835 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mn5ld"] Oct 29 22:48:41.274280 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:41.274248 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" path="/var/lib/kubelet/pods/89b0c9d7-f5e3-408a-ac68-92b6a9a8387c/volumes" Oct 29 22:48:46.433215 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:46.433172 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:48:50.708137 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:50.708094 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 22:48:50.708712 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:50.708359 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerName="maas-api" containerID="cri-o://b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef" gracePeriod=30 Oct 29 22:48:50.793602 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:50.793564 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.58:8443/health\": dial tcp 10.134.0.58:8443: connect: connection refused" Oct 29 22:48:50.971648 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:50.971576 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 22:48:51.133644 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.133591 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\") pod \"3551b155-6d88-4b72-865a-02c0c00fe90a\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " Oct 29 22:48:51.133940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.133731 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\") pod \"3551b155-6d88-4b72-865a-02c0c00fe90a\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " Oct 29 22:48:51.133940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.133766 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\" (UniqueName: \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\") pod \"3551b155-6d88-4b72-865a-02c0c00fe90a\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " Oct 29 22:48:51.133940 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.133857 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\") pod \"3551b155-6d88-4b72-865a-02c0c00fe90a\" (UID: \"3551b155-6d88-4b72-865a-02c0c00fe90a\") " Oct 29 22:48:51.135185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.134217 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca" pod "3551b155-6d88-4b72-865a-02c0c00fe90a" (UID: "3551b155-6d88-4b72-865a-02c0c00fe90a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:51.135185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.134357 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3551b155-6d88-4b72-865a-02c0c00fe90a-openshift-service-ca\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:51.137791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.137743 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls" pod "3551b155-6d88-4b72-865a-02c0c00fe90a" (UID: "3551b155-6d88-4b72-865a-02c0c00fe90a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:51.137791 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.137757 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx" pod "3551b155-6d88-4b72-865a-02c0c00fe90a" (UID: "3551b155-6d88-4b72-865a-02c0c00fe90a"). InnerVolumeSpecName "kube-api-access-7rmnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:51.138160 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.138131 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs" pod "3551b155-6d88-4b72-865a-02c0c00fe90a" (UID: "3551b155-6d88-4b72-865a-02c0c00fe90a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:51.235373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.235289 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-metrics-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:51.235373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.235322 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3551b155-6d88-4b72-865a-02c0c00fe90a-maas-api-tls\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:51.235373 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.235333 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rmnx\" (UniqueName: \"kubernetes.io/projected/3551b155-6d88-4b72-865a-02c0c00fe90a-kube-api-access-7rmnx\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:51.381224 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.381185 4376 generic.go:356] "Generic (PLEG): container finished" podID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerID="b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef" exitCode=0 Oct 29 22:48:51.381424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.381248 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" Oct 29 22:48:51.381424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.381266 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" event={"ID":"3551b155-6d88-4b72-865a-02c0c00fe90a","Type":"ContainerDied","Data":"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef"} Oct 29 22:48:51.381424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.381316 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf" event={"ID":"3551b155-6d88-4b72-865a-02c0c00fe90a","Type":"ContainerDied","Data":"0d67994a74e30307762b519d0d15a6e0ad926f3a3197db62757fb3419b64f534"} Oct 29 22:48:51.381424 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.381339 4376 scope.go:122] "RemoveContainer" containerID="b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef" Oct 29 22:48:51.390045 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.390022 4376 scope.go:122] "RemoveContainer" containerID="b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef" Oct 29 22:48:51.390319 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:48:51.390295 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef\": container with ID starting with b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef not found: ID does not exist" containerID="b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef" Oct 29 22:48:51.390420 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.390327 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef"} err="failed to get container status \"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef\": rpc error: code = NotFound desc = could not find container \"b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef\": container with ID starting with b174dde6ca954d26180c5833bf893ed59a08358018f8e5db6d688cc43a70d2ef not found: ID does not exist" Oct 29 22:48:51.439247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.439207 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 22:48:51.453061 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:51.453028 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-91520c-6fcc7c9f78-4rzcf"] Oct 29 22:48:53.269959 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:53.269923 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" path="/var/lib/kubelet/pods/3551b155-6d88-4b72-865a-02c0c00fe90a/volumes" Oct 29 22:48:59.621095 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:59.621058 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 22:48:59.621889 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:48:59.621852 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" containerID="cri-o://38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b" gracePeriod=30 Oct 29 22:49:00.462274 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:00.462236 4376 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:49:00.462472 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:00.462295 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:49:03.658988 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:03.658947 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:49:04.104918 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.104872 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:49:04.878144 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.878116 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 22:49:04.957513 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957483 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.957715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957531 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.957715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957583 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.957715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957610 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.957715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957659 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.957715 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957712 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.958081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957757 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.958081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957779 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:04.958081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957784 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.958081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.957893 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\") pod \"d4e14c1f-c95e-4af2-b72e-f288fc598938\" (UID: \"d4e14c1f-c95e-4af2-b72e-f288fc598938\") " Oct 29 22:49:04.958081 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958058 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:04.958361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958095 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:04.958361 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:04.958491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958399 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-certs\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:04.958491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958422 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-credential-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:04.958491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958437 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-data\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:04.958491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958451 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e14c1f-c95e-4af2-b72e-f288fc598938-istiod-ca-cert\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:04.958491 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.958464 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:04.960341 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.960312 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:49:04.960462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.960392 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:04.960462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.960396 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:04.960700 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:04.960683 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc" pod "d4e14c1f-c95e-4af2-b72e-f288fc598938" (UID: "d4e14c1f-c95e-4af2-b72e-f288fc598938"). InnerVolumeSpecName "kube-api-access-92hlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:05.059126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.059081 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-workload-socket\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:05.059126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.059121 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-podinfo\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:05.059126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.059131 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-token\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:05.059126 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.059140 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e14c1f-c95e-4af2-b72e-f288fc598938-istio-envoy\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:05.059426 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.059149 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92hlc\" (UniqueName: \"kubernetes.io/projected/d4e14c1f-c95e-4af2-b72e-f288fc598938-kube-api-access-92hlc\") on node \"ip-10-0-141-100.ec2.internal\" DevicePath \"\"" Oct 29 22:49:05.450153 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.450118 4376 generic.go:356] "Generic (PLEG): container finished" podID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerID="38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b" exitCode=0 Oct 29 22:49:05.450353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.450164 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" Oct 29 22:49:05.450353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.450170 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" event={"ID":"d4e14c1f-c95e-4af2-b72e-f288fc598938","Type":"ContainerDied","Data":"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b"} Oct 29 22:49:05.450353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.450219 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h" event={"ID":"d4e14c1f-c95e-4af2-b72e-f288fc598938","Type":"ContainerDied","Data":"7afb3b1879afe594e8ed9d3e882e107f6b908a061c0e2bdeda3796d0da6a8046"} Oct 29 22:49:05.450353 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.450240 4376 scope.go:122] "RemoveContainer" containerID="38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b" Oct 29 22:49:05.459569 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.459540 4376 scope.go:122] "RemoveContainer" containerID="38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b" Oct 29 22:49:05.459896 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:49:05.459871 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b\": container with ID starting with 38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b not found: ID does not exist" containerID="38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b" Oct 29 22:49:05.459990 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.459928 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b"} err="failed to get container status \"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b\": rpc error: code = NotFound desc = could not find container \"38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b\": container with ID starting with 38fd2d26d8c9ef5ccc2c9b429a5bbc888296a7df864c29beb894fd4cbcc68e7b not found: ID does not exist" Oct 29 22:49:05.522582 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.522543 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 22:49:05.536448 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.536375 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-ns68h"] Oct 29 22:49:05.765195 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:05.765159 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:49:07.270386 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:07.270343 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" path="/var/lib/kubelet/pods/d4e14c1f-c95e-4af2-b72e-f288fc598938/volumes" Oct 29 22:49:16.433312 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:49:16.433265 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:49:17.629728 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:17.629690 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:49:18.121758 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:18.121727 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:49:19.748328 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:19.748296 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:49:20.765080 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:20.765041 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85cfbb945d-c2b2m_3ad22e41-a5e2-402f-b9db-4b9138de4b58/maas-api/0.log" Oct 29 22:49:22.845247 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:22.845216 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2dxs2_4f5dc256-d231-44dc-97ef-43c4748be472/postgres/0.log" Oct 29 22:49:27.226322 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:27.226286 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-mls86_0d14731c-01a7-4c2d-8308-d16d6df7cb5c/kube-auth-proxy/0.log" Oct 29 22:49:27.831863 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:27.831827 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-kmrzx_f7ff83b6-0644-419f-9452-ac45a35976ee/istio-proxy/0.log" Oct 29 22:49:28.236011 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:28.233728 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w5vm_b0f77ed1-72e6-425b-9967-ef791198a32b/payload-pre-processing/0.log" Oct 29 22:49:28.426173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:28.426142 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:49:29.574549 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:29.574514 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-22vq4_c837cb92-b1eb-4228-99d1-80b215749ef2/main/0.log" Oct 29 22:49:30.183303 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:30.183267 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-q8tnt_06bc08b1-272a-45fd-981f-69fe88948bc0/main/0.log" Oct 29 22:49:30.386550 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:30.386518 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lcqc5_75921689-7442-45e6-b3ce-8b58c947ba78/main/0.log" Oct 29 22:49:30.592021 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:30.591988 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-xczl2_1ae6e11a-dcdc-47bc-baaf-05f17813bdfa/main/0.log" Oct 29 22:49:41.129228 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129195 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jxpkj/must-gather-4xvwm"] Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129603 4376 cpu_manager.go:414] "containerMap: removing container" podUID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" containerName="discovery" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129615 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" containerName="discovery" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129631 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129637 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129645 4376 cpu_manager.go:414] "containerMap: removing container" podUID="72f864ce-c860-4d20-b75f-35b4bd13cee7" containerName="maas-api" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129650 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f864ce-c860-4d20-b75f-35b4bd13cee7" containerName="maas-api" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129660 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" Oct 29 22:49:41.129662 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129665 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129671 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129675 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129685 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129690 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129697 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c239f9a9-c47e-440c-874f-ad512120b4f2" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129702 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c239f9a9-c47e-440c-874f-ad512120b4f2" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129708 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129713 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129748 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129753 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129759 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129763 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129770 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129774 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129799 4376 cpu_manager.go:414] "containerMap: removing container" podUID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129806 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129818 4376 cpu_manager.go:414] "containerMap: removing container" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129826 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129838 4376 cpu_manager.go:414] "containerMap: removing container" podUID="30bc08dc-cb42-46aa-ac40-ba7c70701234" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129847 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bc08dc-cb42-46aa-ac40-ba7c70701234" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129863 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129870 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129882 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129889 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129899 4376 cpu_manager.go:414] "containerMap: removing container" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129935 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129947 4376 cpu_manager.go:414] "containerMap: removing container" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129953 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129970 4376 cpu_manager.go:414] "containerMap: removing container" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129977 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129987 4376 cpu_manager.go:414] "containerMap: removing container" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.129993 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130001 4376 cpu_manager.go:414] "containerMap: removing container" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130008 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130024 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a401e66c-872c-4d84-bf69-213932b8cd09" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130030 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a401e66c-872c-4d84-bf69-213932b8cd09" containerName="maas-api" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130036 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130043 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130052 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" Oct 29 22:49:41.130029 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130057 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130071 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130077 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130085 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4fc313ec-8b15-48da-9467-c70f2282594f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130092 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc313ec-8b15-48da-9467-c70f2282594f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130102 4376 cpu_manager.go:414] "containerMap: removing container" podUID="842bac1d-a464-4937-8668-72bb7966e6e0" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130108 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842bac1d-a464-4937-8668-72bb7966e6e0" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130117 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130123 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130132 4376 cpu_manager.go:414] "containerMap: removing container" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130139 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130150 4376 cpu_manager.go:414] "containerMap: removing container" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130155 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130162 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130170 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130181 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130186 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130192 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ec83798f-31a3-4d39-8de9-16f11610491f" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130198 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec83798f-31a3-4d39-8de9-16f11610491f" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130206 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130213 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130227 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130233 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c191fe-b715-4094-9b8d-3bd281efd9b8" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130240 4376 cpu_manager.go:414] "containerMap: removing container" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130247 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130257 4376 cpu_manager.go:414] "containerMap: removing container" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130263 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130270 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130275 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130280 4376 cpu_manager.go:414] "containerMap: removing container" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130286 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130296 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130303 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130318 4376 cpu_manager.go:414] "containerMap: removing container" podUID="34b5ca49-5e59-48df-b0a7-838f91298c62" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130328 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b5ca49-5e59-48df-b0a7-838f91298c62" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130350 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130357 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130368 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130375 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130384 4376 cpu_manager.go:414] "containerMap: removing container" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130389 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130396 4376 cpu_manager.go:414] "containerMap: removing container" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130400 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130407 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130412 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130417 4376 cpu_manager.go:414] "containerMap: removing container" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130424 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130437 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a9fff7a2-f346-408c-83ee-b8a591643a9f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130444 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fff7a2-f346-408c-83ee-b8a591643a9f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130451 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130457 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130463 4376 cpu_manager.go:414] "containerMap: removing container" podUID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130468 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130474 4376 cpu_manager.go:414] "containerMap: removing container" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130479 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130485 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130489 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130495 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130500 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130505 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130510 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130517 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130521 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130528 4376 cpu_manager.go:414] "containerMap: removing container" podUID="533d11b2-d4bb-48eb-b462-81732dee78e9" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130533 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533d11b2-d4bb-48eb-b462-81732dee78e9" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130617 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c04004-8e34-43fd-bce0-cf58df2383f4" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130627 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130634 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1e3fd0-9ba4-4d71-904a-1ef68df3b059" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130639 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c239f9a9-c47e-440c-874f-ad512120b4f2" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130645 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d2bc07-2df6-4758-b256-995e638df935" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130652 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e14c1f-c95e-4af2-b72e-f288fc598938" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130661 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd25ab2-2855-46a2-aed0-8b4abfad235e" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130666 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bc3b9e-325c-43c6-9211-0ad5d8fe87c9" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130671 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="814c9c42-6fba-4eea-887a-8d4acc7c26a9" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130678 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="084fee8e-781d-417d-a76b-65b26b25c9d1" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130683 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bc08dc-cb42-46aa-ac40-ba7c70701234" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130689 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e69a718-86ef-4ea7-a475-fc5e7fe62476" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130694 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb09b2e-a108-4f20-8404-a3d06e01e1c4" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130699 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c5c7b6-bfe1-4a6e-8be5-5429e7a24acd" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130705 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f0d211-5adf-40ca-8c85-56878719bf15" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130709 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130715 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130720 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b0d153-a9a9-40fb-9627-02d86d2b68a2" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130726 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a401e66c-872c-4d84-bf69-213932b8cd09" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130731 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19e4df8-c3f6-4acf-95b1-62dfcb501bf3" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130736 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee972f81-b171-40b5-8642-a8f34172e90d" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130742 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8723965e-5757-49b5-a066-865ce2b12406" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130749 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc313ec-8b15-48da-9467-c70f2282594f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130755 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="842bac1d-a464-4937-8668-72bb7966e6e0" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130760 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63ec822-782a-41ee-b6d0-81ae8a852fc3" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130766 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="439a7a25-91d7-4d82-a5bd-93cb94f10eed" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130771 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130776 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cbeffd-8427-4f7b-9c9e-8394f6a12a96" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130782 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1360c02e-3a09-45ca-9d4c-f6241c9c66f0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130787 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec83798f-31a3-4d39-8de9-16f11610491f" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130791 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea6b72b-dd47-4ce5-af72-3b3f9b90f699" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130796 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ec2e5a-2fc1-434a-a023-24016a6333d0" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130802 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130808 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd5fc22-a3d3-4c94-88f5-fea689a73e6f" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130813 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="be560f2c-f950-48ba-97d8-4f2a112d590c" containerName="payload-pre-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130819 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3551b155-6d88-4b72-865a-02c0c00fe90a" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130826 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b5ca49-5e59-48df-b0a7-838f91298c62" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130830 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c364880-627c-428e-8713-6d29733874bd" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130835 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a90d29-40a0-4aa8-9991-61f234b3e273" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130841 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fd011b-f743-4c5e-b61a-a3c62d5ed463" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130845 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="411dd6f0-fe08-41b2-982d-d2cdfe258fad" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130850 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4698b55f-414e-42d3-a92f-ad7ecfeec77e" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130855 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="48adcf8c-be0d-487e-a79a-a4cf4e066126" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130860 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fff7a2-f346-408c-83ee-b8a591643a9f" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130865 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130869 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6b2c47-75c9-4ad7-9090-a2e1a1320fd4" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130875 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa00f6c-e1bd-41b7-96fe-cacd076832d8" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130880 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6d09c5-b567-4547-b8e3-075959fccbe0" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130886 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2fba5c-c2b7-44d1-87fd-12d623f8cc87" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130892 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c354ed1c-da1b-459f-b7b1-4089adb4d83d" containerName="main" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130898 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3443574-dc80-4ca9-b8c1-8c052afaf411" containerName="payload-processing" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130918 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130924 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="533d11b2-d4bb-48eb-b462-81732dee78e9" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130929 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b0c9d7-f5e3-408a-ac68-92b6a9a8387c" containerName="discovery" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130933 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.130938 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f864ce-c860-4d20-b75f-35b4bd13cee7" containerName="maas-api" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.131010 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.131016 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58c4821-b7f6-4b5b-8215-48c20db90b6b" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.131041 4376 cpu_manager.go:414] "containerMap: removing container" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.131045 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51391134-c2b5-4bdf-b54f-971a421cfcaf" containerName="istio-proxy" Oct 29 22:49:41.131462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.131113 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="553a8d1c-4d61-4341-a642-699027f3f480" containerName="istio-proxy" Oct 29 22:49:41.136098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.133970 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r4vts/must-gather-h5ckp"] Oct 29 22:49:41.136098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.134087 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.137395 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.137378 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.153553 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.153505 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4vts/must-gather-h5ckp"] Oct 29 22:49:41.157077 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.157053 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jxpkj/must-gather-4xvwm"] Oct 29 22:49:41.165897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.165870 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jxpkj\"/\"default-dockercfg-z7brt\"" Oct 29 22:49:41.168243 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.168222 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r4vts\"/\"default-dockercfg-brq8c\"" Oct 29 22:49:41.168998 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.168979 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4vts\"/\"kube-root-ca.crt\"" Oct 29 22:49:41.169294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.169258 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jxpkj\"/\"kube-root-ca.crt\"" Oct 29 22:49:41.169600 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.169582 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4vts\"/\"openshift-service-ca.crt\"" Oct 29 22:49:41.169696 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.169682 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jxpkj\"/\"openshift-service-ca.crt\"" Oct 29 22:49:41.221679 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.221653 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbwp5/must-gather-mnc7t"] Oct 29 22:49:41.227203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.227176 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.243000 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.242975 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/must-gather-mnc7t"] Oct 29 22:49:41.256325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.256295 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"openshift-service-ca.crt\"" Oct 29 22:49:41.256458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.256330 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"kube-root-ca.crt\"" Oct 29 22:49:41.256458 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.256411 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gbwp5\"/\"default-dockercfg-ldfvl\"" Oct 29 22:49:41.303639 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.303607 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lts\" (UniqueName: \"kubernetes.io/projected/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-kube-api-access-c6lts\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.303784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.303681 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d863779f-cbb2-44db-bc2a-8f96b20ff484-must-gather-output\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.303784 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.303767 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trx54\" (UniqueName: \"kubernetes.io/projected/d863779f-cbb2-44db-bc2a-8f96b20ff484-kube-api-access-trx54\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.304031 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.304009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-must-gather-output\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.404638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404550 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-must-gather-output\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.404638 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404608 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lts\" (UniqueName: \"kubernetes.io/projected/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-kube-api-access-c6lts\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.404856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404642 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d863779f-cbb2-44db-bc2a-8f96b20ff484-must-gather-output\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.404856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404663 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx54\" (UniqueName: \"kubernetes.io/projected/d863779f-cbb2-44db-bc2a-8f96b20ff484-kube-api-access-trx54\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.404856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmtn\" (UniqueName: \"kubernetes.io/projected/6721c37c-f4d9-4ddf-806b-c568d93f556a-kube-api-access-hgmtn\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.404856 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404712 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6721c37c-f4d9-4ddf-806b-c568d93f556a-must-gather-output\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.405107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.404982 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-must-gather-output\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.405107 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.405053 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d863779f-cbb2-44db-bc2a-8f96b20ff484-must-gather-output\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.431196 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.431168 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lts\" (UniqueName: \"kubernetes.io/projected/da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9-kube-api-access-c6lts\") pod \"must-gather-h5ckp\" (UID: \"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9\") " pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.431316 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.431274 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx54\" (UniqueName: \"kubernetes.io/projected/d863779f-cbb2-44db-bc2a-8f96b20ff484-kube-api-access-trx54\") pod \"must-gather-4xvwm\" (UID: \"d863779f-cbb2-44db-bc2a-8f96b20ff484\") " pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.460686 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.460660 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" Oct 29 22:49:41.469723 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.469702 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4vts/must-gather-h5ckp" Oct 29 22:49:41.505719 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.505680 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmtn\" (UniqueName: \"kubernetes.io/projected/6721c37c-f4d9-4ddf-806b-c568d93f556a-kube-api-access-hgmtn\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.505913 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.505727 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6721c37c-f4d9-4ddf-806b-c568d93f556a-must-gather-output\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.506226 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.506199 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6721c37c-f4d9-4ddf-806b-c568d93f556a-must-gather-output\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.536047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.534204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmtn\" (UniqueName: \"kubernetes.io/projected/6721c37c-f4d9-4ddf-806b-c568d93f556a-kube-api-access-hgmtn\") pod \"must-gather-mnc7t\" (UID: \"6721c37c-f4d9-4ddf-806b-c568d93f556a\") " pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.559203 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.559175 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" Oct 29 22:49:41.766073 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.766045 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jxpkj/must-gather-4xvwm"] Oct 29 22:49:41.768006 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:49:41.767975 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd863779f_cbb2_44db_bc2a_8f96b20ff484.slice/crio-c0e69c9f9977ec7fd07df5b9a3fe203b689ca88e5656abed212db69223c7c619 WatchSource:0}: Error finding container c0e69c9f9977ec7fd07df5b9a3fe203b689ca88e5656abed212db69223c7c619: Status 404 returned error can't find the container with id c0e69c9f9977ec7fd07df5b9a3fe203b689ca88e5656abed212db69223c7c619 Oct 29 22:49:41.769793 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.769773 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:49:41.780774 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.780750 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4vts/must-gather-h5ckp"] Oct 29 22:49:41.782475 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:49:41.782443 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda2ad5a1_4f0a_4a48_b4ea_692cc60bf3c9.slice/crio-d37b88a2fa59e50445ea8017425a1eb1080281e48075c052d79e1100e4574a74 WatchSource:0}: Error finding container d37b88a2fa59e50445ea8017425a1eb1080281e48075c052d79e1100e4574a74: Status 404 returned error can't find the container with id d37b88a2fa59e50445ea8017425a1eb1080281e48075c052d79e1100e4574a74 Oct 29 22:49:41.863430 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:41.863399 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/must-gather-mnc7t"] Oct 29 22:49:41.866776 ip-10-0-141-100 kubenswrapper[4376]: W1029 22:49:41.866751 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6721c37c_f4d9_4ddf_806b_c568d93f556a.slice/crio-107c7760fd8126223260a4608d6919a85664743e865012200bf677eb3efc7e7b WatchSource:0}: Error finding container 107c7760fd8126223260a4608d6919a85664743e865012200bf677eb3efc7e7b: Status 404 returned error can't find the container with id 107c7760fd8126223260a4608d6919a85664743e865012200bf677eb3efc7e7b Oct 29 22:49:42.649800 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:42.649759 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" event={"ID":"6721c37c-f4d9-4ddf-806b-c568d93f556a","Type":"ContainerStarted","Data":"107c7760fd8126223260a4608d6919a85664743e865012200bf677eb3efc7e7b"} Oct 29 22:49:42.653967 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:42.653887 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" event={"ID":"d863779f-cbb2-44db-bc2a-8f96b20ff484","Type":"ContainerStarted","Data":"c0e69c9f9977ec7fd07df5b9a3fe203b689ca88e5656abed212db69223c7c619"} Oct 29 22:49:42.671325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:42.671229 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4vts/must-gather-h5ckp" event={"ID":"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9","Type":"ContainerStarted","Data":"d37b88a2fa59e50445ea8017425a1eb1080281e48075c052d79e1100e4574a74"} Oct 29 22:49:43.690432 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:43.690334 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" event={"ID":"6721c37c-f4d9-4ddf-806b-c568d93f556a","Type":"ContainerStarted","Data":"9acebd01bb19fcd0d846cc61b3ae34b3e9b24de6d70bc85e494c94301772c60b"} Oct 29 22:49:44.726298 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:44.726254 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" event={"ID":"6721c37c-f4d9-4ddf-806b-c568d93f556a","Type":"ContainerStarted","Data":"efabe64d1b0a846bc7f6124983c37bf9f2a930e46240233b045f17d1d468d046"} Oct 29 22:49:44.774937 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:44.774858 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gbwp5/must-gather-mnc7t" podStartSLOduration=2.5326432739999998 podStartE2EDuration="3.774838166s" totalImagesPullingTime="1.242194892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:44.772783725 +0000 UTC m=+4043.621260573" watchObservedRunningTime="2026-10-29 22:49:44.774838166 +0000 UTC m=+4043.623315013" Oct 29 22:49:46.444077 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:49:46.443471 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:49:47.775897 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:47.775861 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z6ll7_96fce558-209a-4a47-a807-af68663c9981/global-pull-secret-syncer/0.log" Oct 29 22:49:48.578592 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:48.575892 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qjxpk_a3b0e867-b455-45f3-94cf-3aeae43cb6ee/konnectivity-agent/0.log" Oct 29 22:49:49.062489 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:49.062448 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-100.ec2.internal_4c9c0102f0b78e22ae4ea8593b59d168/haproxy/0.log" Oct 29 22:49:56.893482 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:56.891444 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" event={"ID":"d863779f-cbb2-44db-bc2a-8f96b20ff484","Type":"ContainerStarted","Data":"673ba294bf28c6e888dde012bbe8cff2c4d936ea06069f8f98dc857f790aefd1"} Oct 29 22:49:56.894053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:56.893989 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4vts/must-gather-h5ckp" event={"ID":"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9","Type":"ContainerStarted","Data":"e55e63ef50227b319ec22664427c07589120b5354d1cf780b95ed48d196dba97"} Oct 29 22:49:56.894053 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:56.894035 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4vts/must-gather-h5ckp" event={"ID":"da2ad5a1-4f0a-4a48-b4ea-692cc60bf3c9","Type":"ContainerStarted","Data":"71108c0ec8edaae1686ec6084aec554b3e30ee3c38e6ba17ca01256b50aabb47"} Oct 29 22:49:56.958173 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:56.958103 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r4vts/must-gather-h5ckp" podStartSLOduration=1.829629685 podStartE2EDuration="15.958082191s" totalImagesPullingTime="14.128452506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:56.952472048 +0000 UTC m=+4055.800948890" watchObservedRunningTime="2026-10-29 22:49:56.958082191 +0000 UTC m=+4055.806559037" Oct 29 22:49:57.913462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:57.913426 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" event={"ID":"d863779f-cbb2-44db-bc2a-8f96b20ff484","Type":"ContainerStarted","Data":"0b4d039fb02ba0c1307d226ba25030b917bad7cab3f80702c4fc7b283c5116c4"} Oct 29 22:49:57.962885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:57.962813 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jxpkj/must-gather-4xvwm" podStartSLOduration=2.181933157 podStartE2EDuration="16.962791821s" totalImagesPullingTime="14.780858664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:57.96026788 +0000 UTC m=+4056.808744724" watchObservedRunningTime="2026-10-29 22:49:57.962791821 +0000 UTC m=+4056.811268670" Oct 29 22:49:59.003669 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.003631 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8"] Oct 29 22:49:59.011683 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.011632 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.026038 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.026009 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8"] Oct 29 22:49:59.136607 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.136563 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-lib-modules\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.136849 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.136835 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-podres\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.136993 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.136978 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-sys\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.137103 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.137093 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftx8w\" (UniqueName: \"kubernetes.io/projected/9ef847e7-2185-44c7-94d8-232f75201e38-kube-api-access-ftx8w\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.137184 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.137175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-proc\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.198956 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.198495 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:49:59.238098 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238027 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-podres\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.238485 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-sys\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.238678 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238649 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-podres\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.238810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-sys\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.238810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftx8w\" (UniqueName: \"kubernetes.io/projected/9ef847e7-2185-44c7-94d8-232f75201e38-kube-api-access-ftx8w\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.238810 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-proc\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.239055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238859 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-proc\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.239055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238874 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-lib-modules\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.239055 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.238965 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef847e7-2185-44c7-94d8-232f75201e38-lib-modules\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.266462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.266382 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftx8w\" (UniqueName: \"kubernetes.io/projected/9ef847e7-2185-44c7-94d8-232f75201e38-kube-api-access-ftx8w\") pod \"perf-node-gather-daemonset-24db8\" (UID: \"9ef847e7-2185-44c7-94d8-232f75201e38\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.330315 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.330275 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.416923 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.415722 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:49:59.507230 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.507168 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:49:59.657209 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.657088 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8"] Oct 29 22:49:59.734922 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.734072 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:49:59.937970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.936288 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" event={"ID":"9ef847e7-2185-44c7-94d8-232f75201e38","Type":"ContainerStarted","Data":"c14ef1800d06a2c5aecafdac2ec036360b1c1a878ef00d706d4a640761d3dded"} Oct 29 22:49:59.937970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.936356 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" event={"ID":"9ef847e7-2185-44c7-94d8-232f75201e38","Type":"ContainerStarted","Data":"93398a8d95d278d9c112bd3ea7133edbbae7eaf7681f3c29d21fa4faaf59e020"} Oct 29 22:49:59.937970 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.937201 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:49:59.987387 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:49:59.987074 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" podStartSLOduration=1.987053086 podStartE2EDuration="1.987053086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:59.982417953 +0000 UTC m=+4058.830894813" watchObservedRunningTime="2026-10-29 22:49:59.987053086 +0000 UTC m=+4058.835529932" Oct 29 22:50:00.797920 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:00.797863 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/alertmanager/0.log" Oct 29 22:50:00.917266 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:00.917229 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/config-reloader/0.log" Oct 29 22:50:01.037626 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.037596 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/kube-rbac-proxy-web/0.log" Oct 29 22:50:01.166338 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.166244 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/kube-rbac-proxy/0.log" Oct 29 22:50:01.309783 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.309749 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/kube-rbac-proxy-metric/0.log" Oct 29 22:50:01.431798 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.431715 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/prom-label-proxy/0.log" Oct 29 22:50:01.555418 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.555374 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8eefb702-9336-4ab0-aa4d-8a584b1a1d4f/init-config-reloader/0.log" Oct 29 22:50:01.853389 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.853358 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fgt5_b228e418-7c35-4d4d-b7b1-67fdaa60b43c/kube-state-metrics/0.log" Oct 29 22:50:01.967506 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:01.967473 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fgt5_b228e418-7c35-4d4d-b7b1-67fdaa60b43c/kube-rbac-proxy-main/0.log" Oct 29 22:50:02.088139 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.088106 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2fgt5_b228e418-7c35-4d4d-b7b1-67fdaa60b43c/kube-rbac-proxy-self/0.log" Oct 29 22:50:02.219540 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.219510 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cb667f678-wvsbj_2f8ca511-4b4b-4448-a680-21c4cffbc095/metrics-server/0.log" Oct 29 22:50:02.353801 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.353764 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wjjc6_a6e75a2e-b02d-4fc2-b60c-6c7d085e7a00/monitoring-plugin/0.log" Oct 29 22:50:02.575740 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.575661 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:02.870121 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.870038 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tnw_736d51e6-a9df-4c00-829e-cc256fdef7b6/node-exporter/0.log" Oct 29 22:50:02.932250 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.932222 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:02.991620 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:02.991250 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tnw_736d51e6-a9df-4c00-829e-cc256fdef7b6/kube-rbac-proxy/0.log" Oct 29 22:50:03.119276 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:03.119249 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59tnw_736d51e6-a9df-4c00-829e-cc256fdef7b6/init-textfile/0.log" Oct 29 22:50:03.575617 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:03.575585 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rpt8d_06fb2d9b-8ddf-4189-acae-554a0dda925d/kube-rbac-proxy-main/0.log" Oct 29 22:50:03.697733 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:03.697703 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rpt8d_06fb2d9b-8ddf-4189-acae-554a0dda925d/kube-rbac-proxy-self/0.log" Oct 29 22:50:03.821158 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:03.821124 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rpt8d_06fb2d9b-8ddf-4189-acae-554a0dda925d/openshift-state-metrics/0.log" Oct 29 22:50:04.123514 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.123481 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:50:04.295659 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.295624 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:50:04.488984 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.488940 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:50:04.690350 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.690308 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:50:04.817185 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.817100 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zhnq8_0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8/prometheus-operator/0.log" Oct 29 22:50:04.928047 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:04.928008 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zhnq8_0b8d9ea6-2ace-4d4c-8b19-2d3289dd44e8/kube-rbac-proxy/0.log" Oct 29 22:50:05.055294 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:05.055254 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-z85qr_4fe6c154-5f39-4b6d-922a-3b63fdda4b77/prometheus-operator-admission-webhook/0.log" Oct 29 22:50:05.184030 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:05.183940 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bc8cf59b7-2kzcq_7c2b63ec-bb25-4889-a94f-473fc2a92cbb/telemeter-client/0.log" Oct 29 22:50:05.304242 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:05.304180 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bc8cf59b7-2kzcq_7c2b63ec-bb25-4889-a94f-473fc2a92cbb/reload/0.log" Oct 29 22:50:05.423864 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:05.423817 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bc8cf59b7-2kzcq_7c2b63ec-bb25-4889-a94f-473fc2a92cbb/kube-rbac-proxy/0.log" Oct 29 22:50:06.961156 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:06.961129 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-24db8" Oct 29 22:50:07.818325 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:07.818289 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:08.004114 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:08.004080 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:09.843462 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:09.843428 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:50:09.880613 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:09.880580 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54b58f7c85-hgxhj_f9cfc987-b018-4040-bdb2-8dc46d06dedb/manager/0.log" Oct 29 22:50:10.092405 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:10.092369 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:50:10.126308 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:10.126209 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-45zz7_f387d9bd-6ff0-4295-9da1-399a1e2d6b1f/manager/0.log" Oct 29 22:50:10.819867 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:10.819831 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57d9b6fb66-57pvx_ac0eddce-c9cd-469c-8b86-37b80e4a5bca/console/0.log" Oct 29 22:50:10.972572 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:10.972539 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-sh62q_7ba8c45b-b622-482b-960c-9fdcfbdde069/download-server/0.log" Oct 29 22:50:13.704474 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:13.704439 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:13.711466 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:13.711432 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-79xrr_7966cd6d-a377-4b75-b61e-d74cc0dd5021/proxy/0.log" Oct 29 22:50:15.635885 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:15.635803 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhzlq_fcb75346-5362-4c07-9564-ba9344456020/dns/0.log" Oct 29 22:50:15.751290 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:15.751254 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhzlq_fcb75346-5362-4c07-9564-ba9344456020/kube-rbac-proxy/0.log" Oct 29 22:50:15.879467 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:15.879435 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2cmw5_b57d5263-1744-42ca-ae20-2c52ab13ba81/dns-node-resolver/0.log" Oct 29 22:50:16.235329 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:16.235291 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-mls86_0d14731c-01a7-4c2d-8308-d16d6df7cb5c/kube-auth-proxy/0.log" Oct 29 22:50:16.414113 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:16.414042 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-kmrzx_f7ff83b6-0644-419f-9452-ac45a35976ee/istio-proxy/0.log" Oct 29 22:50:16.439331 ip-10-0-141-100 kubenswrapper[4376]: E1029 22:50:16.439257 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhgtx" containerName="extractor" Oct 29 22:50:16.567665 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:16.567572 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w5vm_b0f77ed1-72e6-425b-9967-ef791198a32b/payload-pre-processing/0.log" Oct 29 22:50:16.640132 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:16.640049 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6cb7b_aaa82eab-67c3-48b0-9141-dacf6d9d4262/payload-processing/0.log" Oct 29 22:50:17.585536 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:17.585501 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gstvd_295393c7-fd50-401c-a65b-95293cc4953a/node-ca/0.log" Oct 29 22:50:18.677927 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:18.677337 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-mls86_0d14731c-01a7-4c2d-8308-d16d6df7cb5c/kube-auth-proxy/0.log" Oct 29 22:50:18.709197 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:18.709054 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-mls86_0d14731c-01a7-4c2d-8308-d16d6df7cb5c/kube-auth-proxy/0.log" Oct 29 22:50:18.875947 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:18.875859 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-kmrzx_f7ff83b6-0644-419f-9452-ac45a35976ee/istio-proxy/0.log" Oct 29 22:50:18.914991 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:18.914950 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-kmrzx_f7ff83b6-0644-419f-9452-ac45a35976ee/istio-proxy/0.log" Oct 29 22:50:19.024261 ip-10-0-141-100 kubenswrapper[4376]: I1029 22:50:19.024220 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w5vm_b0f77ed1-72e6-425b-9967-ef791198a32b/payload-pre-processing/0.log"